[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479880688 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.032 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002404] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008334] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 0.010016] Calibrating delay loop (skipped) preset value.. 4800.06 BogoMIPS (lpj=2400032) [ 0.011013] pid_max: default: 32768 minimum: 301 [ 0.012150] LSM: Security Framework initializing [ 0.013069] Yama: becoming mindful. [ 0.014045] SELinux: Initializing. [ 0.015083] *** VALIDATE selinux *** [ 0.024102] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028838] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031112] *** VALIDATE tmpfs *** [ 0.032441] *** VALIDATE proc *** [ 0.033286] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037005] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040033] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.043872] debug: unmapping init [mem 0xffffffffa5659000-0xffffffffa5660fff] [ 0.045880] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046682] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050008] ... value mask: 0000ffffffffffff [ 0.051010] ... max period: 00007fffffffffff [ 0.052011] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054284] rcu: Hierarchical SRCU implementation. [ 0.056247] smp: Bringing up secondary CPUs ... [ 0.057586] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.061378] smp: Brought up 1 node, 4 CPUs [ 0.063011] smpboot: Max logical packages: 1 [ 0.063993] smpboot: Total of 4 processors activated (19200.25 BogoMIPS) [ 0.140013] node 0 deferred pages initialised in 75ms [ 0.142145] devtmpfs: initialized [ 0.143160] x86/mm: Memory block size: 128MB [ 0.145691] gcov: version magic: 0x41383552 [ 0.146593] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152368] pinctrl core: initialized pinctrl subsystem [ 0.154172] [ 0.154771] ************************************************************* [ 0.158016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160011] ** ** [ 0.163014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165011] ** ** [ 0.168012] ** This means that this kernel is built to expose internal ** [ 0.169008] ** IOMMU data structures, which may compromise security on ** [ 0.171014] ** your system. ** [ 0.174011] ** ** [ 0.176010] ** If you see this message and you are not debugging the ** [ 0.178015] ** kernel, report this immediately to your vendor! ** [ 0.181011] ** ** [ 0.183009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185012] ************************************************************* [ 0.187821] NET: Registered protocol family 16 [ 0.189418] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198010] cpuidle: using governor menu [ 0.199642] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202718] PCI: Using configuration type 1 for base access [ 0.205135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216105] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220066] cryptd: max_cpu_qlen set to 1000 [ 0.224037] ACPI: Added _OSI(Module Device) [ 0.225016] ACPI: Added _OSI(Processor Device) [ 0.226031] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.227055] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233106] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.239657] ACPI: Interpreter enabled [ 0.242074] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243010] ACPI: Using IOAPIC for interrupt routing [ 0.245101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247371] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257483] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268231] acpiphp: Slot [2] registered [ 0.270168] acpiphp: Slot [5] registered [ 0.272156] acpiphp: Slot [6] registered [ 0.273126] acpiphp: Slot [7] registered [ 0.275151] acpiphp: Slot [8] registered [ 0.276112] acpiphp: Slot [9] registered [ 0.277097] acpiphp: Slot [10] registered [ 0.279130] acpiphp: Slot [3] registered [ 0.280087] acpiphp: Slot [4] registered [ 0.282106] acpiphp: Slot [11] registered [ 0.283098] acpiphp: Slot [12] registered [ 0.284104] acpiphp: Slot [13] registered [ 0.286090] acpiphp: Slot [14] registered [ 0.288160] acpiphp: Slot [15] registered [ 0.289085] acpiphp: Slot [16] registered [ 0.290068] acpiphp: Slot [17] registered [ 0.291019] acpiphp: Slot [18] registered [ 0.292101] acpiphp: Slot [19] registered [ 0.294096] acpiphp: Slot [20] registered [ 0.295121] acpiphp: Slot [21] registered [ 0.297141] acpiphp: Slot [22] registered [ 0.299113] acpiphp: Slot [23] registered [ 0.300165] acpiphp: Slot [24] registered [ 0.302102] acpiphp: Slot [25] registered [ 0.303116] acpiphp: Slot [26] registered [ 0.305096] acpiphp: Slot [27] registered [ 0.306125] acpiphp: Slot [28] registered [ 0.308097] acpiphp: Slot [29] registered [ 0.309118] acpiphp: Slot [30] registered [ 0.310102] acpiphp: Slot [31] registered [ 0.312120] PCI host bridge to bus 0000:00 [ 0.313016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.315022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.317019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.320023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.322024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.325022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.326179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.329286] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.332400] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.342856] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.347041] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.357649] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.360610] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.366911] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.373018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.384017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.389905] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.395787] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.404018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.412021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.430022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.441149] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.449027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.455025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.471019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.478062] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.485022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.492023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.510024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.521271] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.526017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.532018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.546019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.555685] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.560016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.566024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.578018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.587683] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.594020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.599022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.615039] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.626279] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.629503] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.632383] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.635436] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.638239] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.644140] iommu: Default domain type: Passthrough [ 0.645437] SCSI subsystem initialized [ 0.647203] ACPI: bus type USB registered [ 0.649163] usbcore: registered new interface driver usbfs [ 0.651142] usbcore: registered new interface driver hub [ 0.653115] usbcore: registered new device driver usb [ 0.655243] pps_core: LinuxPPS API ver. 1 registered [ 0.657017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.660088] PTP clock support registered [ 0.662112] EDAC MC: Ver: 3.0.0 [ 0.664150] PCI: Using ACPI for IRQ routing [ 0.665896] NetLabel: Initializing [ 0.669017] NetLabel: domain hash size = 128 [ 0.671025] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.673096] NetLabel: unlabeled traffic allowed by default [ 0.676171] vgaarb: loaded [ 0.677348] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.679026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.686412] clocksource: Switched to clocksource kvm-clock [ 0.795663] VFS: Disk quotas dquot_6.6.0 [ 0.797762] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.799741] *** VALIDATE ramfs *** [ 0.800737] *** VALIDATE hugetlbfs *** [ 0.801775] pnp: PnP ACPI init [ 0.804297] pnp: PnP ACPI: found 6 devices [ 0.822252] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.825378] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.827015] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.828530] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.830279] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.832916] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.835414] NET: Registered protocol family 2 [ 0.837810] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.842967] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.847453] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.853619] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.868812] TCP: Hash tables configured (established 65536 bind 65536) [ 0.874955] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.878116] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.881035] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.884475] NET: Registered protocol family 1 [ 0.888250] RPC: Registered named UNIX socket transport module. [ 0.890483] RPC: Registered udp transport module. [ 0.892767] RPC: Registered tcp transport module. [ 0.895090] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.898047] NET: Registered protocol family 44 [ 0.899722] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.902128] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.904692] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.907246] PCI: CLS 0 bytes, default 64 [ 0.909218] Unpacking initramfs... [ 2.322763] debug: unmapping init [mem 0xffff996efcc54000-0xffff996efffbffff] [ 2.327519] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.330015] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.333441] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 2.834701] Initialise system trusted keyrings [ 2.836602] Key type blacklist registered [ 2.838900] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.848257] zbud: loaded [ 2.851367] *** VALIDATE nfs *** [ 2.852849] *** VALIDATE nfs4 *** [ 2.854619] pstore: using deflate compression [ 2.858364] Platform Keyring initialized [ 2.954252] NET: Registered protocol family 38 [ 2.955985] Key type asymmetric registered [ 2.957862] Asymmetric key parser 'x509' registered [ 2.959353] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.962786] io scheduler mq-deadline registered [ 2.964308] io scheduler kyber registered [ 2.966301] io scheduler bfq registered [ 2.968532] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.972168] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.975790] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.979316] ACPI: Power Button [PWRF] [ 2.984051] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.990613] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.003873] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.011669] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.028396] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.055684] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.093996] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.117344] Non-volatile memory driver v1.3 [ 3.119147] Linux agpgart interface v0.103 [ 3.158935] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.163350] vda: detected capacity change from 0 to 34836480 [ 3.180597] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.184523] vdb: detected capacity change from 0 to 1073741824 [ 3.201399] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.204374] vdc: detected capacity change from 0 to 2621440000 [ 3.219205] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.222794] vdd: detected capacity change from 0 to 2621440000 [ 3.238632] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.242102] vde: detected capacity change from 0 to 4294967296 [ 3.259593] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.262888] vdf: detected capacity change from 0 to 4294967296 [ 3.268297] libphy: Fixed MDIO Bus: probed [ 3.273868] usbcore: registered new interface driver usbserial_generic [ 3.276369] usbserial: USB Serial support registered for generic [ 3.279032] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.283365] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.285834] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.290107] mousedev: PS/2 mouse device common for all mice [ 3.293611] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.294535] rtc_cmos 00:05: RTC can wake from S4 [ 3.300505] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.304729] rtc_cmos 00:05: registered as rtc0 [ 3.307995] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.309654] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.312535] intel_pstate: CPU model not supported [ 3.319243] hid: raw HID events driver (C) Jiri Kosina [ 3.321774] usbcore: registered new interface driver usbhid [ 3.324176] usbhid: USB HID core driver [ 3.326242] drop_monitor: Initializing network drop monitor service [ 3.329222] Initializing XFRM netlink socket [ 3.331611] NET: Registered protocol family 10 [ 3.334787] Segment Routing with IPv6 [ 3.336454] NET: Registered protocol family 17 [ 3.338762] mpls_gso: MPLS GSO support [ 3.344960] RAS: Correctable Errors collector initialized. [ 3.347830] AVX version of gcm_enc/dec engaged. [ 3.349814] AES CTR mode by8 optimization enabled [ 3.439564] sched_clock: Marking stable (3439541070, 0)->(4392387490, -952846420) [ 3.443460] registered taskstats version 1 [ 3.445783] Loading compiled-in X.509 certificates [ 3.447706] zswap: loaded using pool lzo/zbud [ 3.470320] Key type big_key registered [ 3.482680] Key type encrypted registered [ 3.484608] ima: No TPM chip found, activating TPM-bypass! [ 3.487063] ima: Allocated hash algorithm: sha1 [ 3.489334] ima: No architecture policies found [ 3.490920] evm: Initialising EVM extended attributes: [ 3.493102] evm: security.selinux [ 3.494167] evm: security.ima [ 3.495222] evm: security.capability [ 3.496491] evm: HMAC attrs: 0x1 [ 3.498825] rtc_cmos 00:05: setting system clock to 2026-04-14 17:49:22 UTC (1776188962) [ 3.505140] debug: unmapping init [mem 0xffffffffa6603000-0xffffffffa67fffff] [ 3.508329] debug: unmapping init [mem 0xffffffffa5382000-0xffffffffa5658fff] [ 3.521130] Write protecting the kernel read-only data: 28672k [ 3.524440] debug: unmapping init [mem 0xffffffffa3a03000-0xffffffffa3bfffff] [ 3.527630] debug: unmapping init [mem 0xffffffffa4314000-0xffffffffa43fffff] [ 3.567949] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.578194] systemd[1]: Detected virtualization kvm. [ 3.580490] systemd[1]: Detected architecture x86-64. [ 3.582489] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.613863] systemd[1]: No hostname configured. [ 3.616713] systemd[1]: Set hostname to . [ 3.618956] random: systemd: uninitialized urandom read (16 bytes read) [ 3.621874] systemd[1]: Initializing machine ID from random generator. [ 3.762607] random: systemd: uninitialized urandom read (16 bytes read) [ 3.765462] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.770522] random: systemd: uninitialized urandom read (16 bytes read) [ 3.773941] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.779118] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.366333] device-mapper: uevent: version 1.0.3 [ 4.369306] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.057779] random: fast init done [ 5.081567] virtio_net virtio0 ens2: renamed from eth0 [ 5.118475] scsi host0: ata_piix [ 5.222484] scsi host1: ata_piix [ 5.224500] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.226898] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.930174] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.014318] random: crng init done [ 10.016105] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.394571] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.439488] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.706347] SELinux: Disabled at runtime. [ 11.765374] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.775267] systemd[1]: Detected virtualization kvm. [ 11.776491] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.279648] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.283688] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.288731] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.294387] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.298355] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.306438] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.318729] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice[ 12.394103] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.751642] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.044189] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.083656] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.161043] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.170619] EDAC sbridge: Ver: 1.1.2 [ 14.876962] Key type dns_resolver registered [ 15.168598] NFS: Registering the id_resolver key type [ 15.170603] Key type id_resolver registered [ 15.172290] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg315-server login: [ 43.436479] libcfs: loading out-of-tree module taints kernel. [ 43.515967] alg: No test for adler32 (adler32-zlib) [ 44.271463] Key type ._llcrypt registered [ 44.273367] Key type .llcrypt registered [ 44.348638] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 61.958061] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 63.612079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.430265] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 65.225111] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 65.230720] LNet: Accept secure, port 988 [ 67.047135] Key type lgssc registered [ 68.893658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.726402] hrtimer: interrupt took 3506025 ns [ 85.237518] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 87.814582] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.537774] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 105.470965] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.695330] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 127.316397] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 138.869628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.939678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 140.203951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 140.236495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 140.329599] Lustre: lustre-MDT0000: new disk, initializing [ 140.418370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 140.432517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.107184] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 157.242869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.324030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.398121] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 157.421466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 157.427483] Lustre: Skipped 1 previous similar message [ 157.561668] Lustre: lustre-MDT0001: new disk, initializing [ 157.702179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.746148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 157.764569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 162.139204] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 175.271575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 175.369970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 175.376657] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 175.694866] Lustre: lustre-OST0000: new disk, initializing [ 175.703606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 175.830645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.624202] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 182.391685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 182.431396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 197.497032] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 197.646069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.651467] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 197.780156] Lustre: lustre-OST0001: new disk, initializing [ 197.796608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 197.875719] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.218960] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 205.892764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 205.905981] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.055804] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 223.444126] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 232.804531] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.996785] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing yml_node [ 242.048648] Lustre: DEBUG MARKER: Client: 2.15.8.11 [ 244.121915] Lustre: DEBUG MARKER: MDS: 2.15.8.11 [ 246.279910] Lustre: DEBUG MARKER: OSS: 2.15.8.11 [ 247.952359] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 13:53:26 EDT 2026 [ 256.292698] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 257.706595] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 272.874014] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 272.874782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 272.891844] Lustre: Skipped 3 previous similar messages [ 272.901089] Lustre: Skipped 2 previous similar messages [ 277.987602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 277.992280] Lustre: Skipped 2 previous similar messages [ 278.218346] Lustre: server umount lustre-MDT0000 complete [ 281.476048] LustreError: 6238:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189240 with bad export cookie 1835486864097856669 [ 281.484573] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 281.496563] LustreError: 6238:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 283.105068] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.115286] Lustre: Skipped 1 previous similar message [ 283.120908] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 283.121495] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 283.143591] LustreError: Skipped 1 previous similar message [ 283.155431] Lustre: Skipped 1 previous similar message [ 286.112162] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 286.112218] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 286.130539] LustreError: Skipped 1 previous similar message [ 286.139367] Lustre: Skipped 1 previous similar message [ 287.140680] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.163574] LustreError: Skipped 1 previous similar message [ 287.894223] Lustre: server umount lustre-MDT0001 complete [ 291.575150] Lustre: server umount lustre-OST0000 complete [ 295.167922] Lustre: server umount lustre-OST0001 complete [ 301.575208] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 308.626800] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 316.939243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 321.749929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 326.610719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 331.469786] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 336.582694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 336.648429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 336.839116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 336.862309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 336.933752] Lustre: lustre-MDT0000: new disk, initializing [ 337.005178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 337.027124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 340.203263] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 350.108729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 350.174516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.285349] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 350.339486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 350.345147] Lustre: Skipped 1 previous similar message [ 350.442510] Lustre: lustre-MDT0001: new disk, initializing [ 350.552272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 350.584836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 350.592859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 354.920150] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 363.440363] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 368.445684] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 374.728065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 374.777266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 374.788291] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 374.951266] Lustre: lustre-OST0000: new disk, initializing [ 374.954869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 375.034320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.291244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 376.303032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 378.504630] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 386.567522] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 390.106215] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 390.284344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 394.078916] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 394.271122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 396.768987] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 396.775784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.787788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 396.799360] Lustre: Skipped 1 previous similar message [ 396.812024] Lustre: Skipped 3 previous similar messages [ 402.166901] Lustre: server umount lustre-OST0000 complete [ 407.009747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 407.014985] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 407.015830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 407.015837] Lustre: Skipped 2 previous similar messages [ 407.026067] Lustre: Skipped 1 previous similar message [ 407.039820] LustreError: Skipped 1 previous similar message [ 412.132201] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.153768] LustreError: Skipped 1 previous similar message [ 413.816435] Lustre: server umount lustre-MDT0000 complete [ 417.173465] LustreError: 14324:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189376 with bad export cookie 1835486864097858125 [ 417.173957] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 417.188689] LustreError: 14324:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 421.161656] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:56:19 (1776189379) [ 423.382986] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 425.028783] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:56:23 (1776189383) [ 431.260282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.741560] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 431.862408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.941969] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 442.338240] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 442.371405] LustreError: Skipped 1 previous similar message [ 445.286301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.772297] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 457.529593] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 462.524283] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 467.772227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 471.708242] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 478.806106] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 493.029353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.055671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.062753] Lustre: Skipped 2 previous similar messages [ 497.207232] Lustre: server umount lustre-OST0000 complete [ 497.209537] Lustre: Skipped 1 previous similar message [ 502.240438] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 502.247680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.269691] Lustre: Skipped 1 previous similar message [ 508.391718] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.408867] LustreError: Skipped 1 previous similar message [ 510.190829] LustreError: 18863:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189469 with bad export cookie 1835486864097858979 [ 510.199426] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.215426] LustreError: 18863:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 521.195572] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 523.977155] Key type lgssc unregistered [ 524.254863] LNet: 22557:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 524.264582] LNet: Removed LNI 192.168.203.115@tcp [ 525.098326] Key type .llcrypt unregistered [ 525.102218] Key type ._llcrypt unregistered [ 544.356434] alg: No test for adler32 (adler32-zlib) [ 545.110359] Key type ._llcrypt registered [ 545.112496] Key type .llcrypt registered [ 545.203280] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 546.146468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 546.620375] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 546.897679] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 546.904553] LNet: Accept secure, port 988 [ 548.623265] Key type lgssc registered [ 549.552941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 557.126886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.729390] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.805343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.195324] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.289161] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 569.314077] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.162773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.506101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 577.650928] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 585.001882] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 590.871296] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 597.501834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 597.748776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.477223] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 609.400511] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 610.033117] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 624.611250] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 624.620437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.643178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 625.634845] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 625.640410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.643900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 625.656929] Lustre: Skipped 1 previous similar message [ 629.693302] Lustre: server umount lustre-OST0000 complete [ 634.848395] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 634.856682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.883810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.936997] Lustre: server umount lustre-MDT0000 complete [ 640.993106] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.010463] LustreError: Skipped 1 previous similar message [ 645.075053] LustreError: 24009:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189604 with bad export cookie 3855646486052816364 [ 645.085684] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 645.086892] LustreError: 24009:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 645.448530] Lustre: server umount lustre-MDT0001 complete [ 656.963376] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 659.369732] Key type lgssc unregistered [ 659.660544] LNet: 27707:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.667213] LNet: Removed LNI 192.168.203.115@tcp [ 660.248511] Key type .llcrypt unregistered [ 660.254787] Key type ._llcrypt unregistered [ 682.101877] alg: No test for adler32 (adler32-zlib) [ 682.858969] Key type ._llcrypt registered [ 682.863873] Key type .llcrypt registered [ 683.003795] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 683.933956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 684.393345] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 684.585666] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 684.589433] LNet: Accept secure, port 988 [ 686.271132] Key type lgssc registered [ 687.367670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 695.454851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.960188] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 697.050801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.100240] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.116952] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 706.530501] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.651750] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.749107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.176727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 716.148470] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 723.148662] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 728.494708] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 734.902733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 735.248392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 739.404480] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 746.546766] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 746.833699] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 752.772130] Lustre: server umount lustre-MDT0000 complete [ 753.125237] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 753.135129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 753.149313] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 756.289167] LustreError: 29165:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189715 with bad export cookie 3222089796028123694 [ 756.289250] LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 [ 756.308909] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 756.705590] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.712429] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 756.728366] Lustre: Skipped 1 previous similar message [ 758.956181] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 760.972391] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 761.825260] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 761.849358] LustreError: Skipped 4 previous similar messages [ 762.694880] Lustre: server umount lustre-MDT0001 complete [ 784.145329] Lustre: server umount lustre-OST0000 complete [ 802.367857] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 805.463434] Key type lgssc unregistered [ 805.795283] LNet: 33179:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.801339] LNet: Removed LNI 192.168.203.115@tcp [ 806.821186] Key type .llcrypt unregistered [ 806.828517] Key type ._llcrypt unregistered [ 828.630151] alg: No test for adler32 (adler32-zlib) [ 829.398123] Key type ._llcrypt registered [ 829.401527] Key type .llcrypt registered [ 829.536249] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 830.597868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 831.110723] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 831.328911] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 831.339213] LNet: Accept secure, port 988 [ 832.983138] Key type lgssc registered [ 834.039346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 843.230226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.858339] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 844.933315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.252215] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 850.403878] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 855.538969] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 859.496550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.812246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 864.486345] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 872.860984] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 878.854727] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 886.128946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 886.583425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 891.520639] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 895.826049] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 899.232899] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 905.879684] Lustre: lustre-OST0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 906.212519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.214751] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 906.231526] Lustre: Skipped 1 previous similar message [ 910.973774] Lustre: lustre-OST0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 910.990671] Lustre: Skipped 2 previous similar messages [ 911.763983] Lustre: server umount lustre-OST0000 complete [ 931.298446] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 931.317310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.345834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 931.349906] Lustre: Skipped 2 previous similar messages [ 935.743316] Lustre: server umount lustre-MDT0000 complete [ 937.959215] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.978067] LustreError: Skipped 1 previous similar message [ 939.806729] LustreError: 34622:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189898 with bad export cookie 15234533357431941019 [ 939.812468] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 939.825520] LustreError: 34622:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 954.335834] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 954.633720] Lustre: server umount lustre-MDT0001 complete [ 966.457602] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 969.951784] Key type lgssc unregistered [ 970.339668] LNet: 38478:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.359863] LNet: Removed LNI 192.168.203.115@tcp [ 971.596714] Key type .llcrypt unregistered [ 971.602173] Key type ._llcrypt unregistered [ 995.040140] alg: No test for adler32 (adler32-zlib) [ 995.823314] Key type ._llcrypt registered [ 995.825886] Key type .llcrypt registered [ 995.974728] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 997.011384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 997.574399] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 997.881678] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 997.890370] LNet: Accept secure, port 988 [ 999.615125] Key type lgssc registered [ 1000.587789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1007.893707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.353685] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1009.423584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.040261] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1014.753922] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1019.877207] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.684669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.999343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1026.314642] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1033.671383] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1038.415302] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1044.810756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1045.110211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.369631] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1056.653192] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1059.467316] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 1066.111835] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1069.373209] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1073.651420] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1073.660734] LNetError: 39242:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1073.665595] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1081.440724] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-1414h07m19s [ 1082.689972] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1085.409754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.412790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1085.420431] Lustre: Skipped 2 previous similar messages [ 1085.427955] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1085.432772] Lustre: Skipped 2 previous similar messages [ 1085.445490] LNetError: 39239:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1085.450989] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1086.096306] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 1090.534108] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1090.539591] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1090.553263] LustreError: Skipped 2 previous similar messages [ 1090.566314] LNetError: 39240:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1090.580281] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1090.580562] Lustre: server umount lustre-MDT0000 complete [ 1093.891973] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1095.649527] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.655322] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1095.672965] LustreError: Skipped 3 previous similar messages [ 1095.685774] LNetError: 39241:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1095.690531] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1097.632088] Lustre: 39253:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776190049/real 1776190049] req@0000000083c989f4 x1862470159322112/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776190056 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1097.674762] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1100.120458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.842125] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.848926] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0x7c6029d12158aafc to 0x7c6029d12158af5c [ 1103.870964] LustreError: Skipped 11 previous similar messages [ 1103.901786] Lustre: MGC192.168.203.115@tcp: Connection restored to 192.168.203.115@tcp (at 0@lo) [ 1104.044808] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1104.058399] LNetError: 39242:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1104.066797] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1104.143752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.215498] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:225 [ 1107.999711] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1109.484762] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1109.500601] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1109.504935] Lustre: Skipped 1 previous similar message [ 1110.935430] Lustre: DEBUG MARKER: Wait for df (28949) ... [ 1112.273487] Lustre: DEBUG MARKER: done [ 1118.178248] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1118.192215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.209541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1119.720196] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1119.728124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.835438] Lustre: server umount lustre-OST0000 complete [ 1129.952818] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1129.961797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.973325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.993171] Lustre: Skipped 1 previous similar message [ 1129.998368] Lustre: Skipped 2 previous similar messages [ 1130.021988] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1130.028782] LNetError: 39239:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1130.035588] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1131.409482] Lustre: server umount lustre-MDT0000 complete [ 1134.696188] LustreError: 39920:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190093 with bad export cookie 8962209236651650908 [ 1134.703506] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1134.707129] LustreError: 39920:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1134.989727] Lustre: server umount lustre-MDT0001 complete [ 1145.083426] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1147.447576] Key type lgssc unregistered [ 1147.697208] LNet: 45047:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.719153] LNet: Removed LNI 192.168.203.115@tcp [ 1148.527872] Key type .llcrypt unregistered [ 1148.531923] Key type ._llcrypt unregistered [ 1178.588150] alg: No test for adler32 (adler32-zlib) [ 1179.345113] Key type ._llcrypt registered [ 1179.349773] Key type .llcrypt registered [ 1179.450556] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 1190.576493] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1191.702268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1192.000152] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 1192.177884] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1192.184501] LNet: Accept secure, port 988 [ 1193.823142] Key type lgssc registered [ 1194.674224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1201.804748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1206.970432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1212.380197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1218.077597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1224.019484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1224.102541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.313765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1225.347432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1225.432568] Lustre: lustre-MDT0000: new disk, initializing [ 1225.510460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.526666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1228.901379] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1236.997977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1237.083435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.145829] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1237.187963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1237.191855] Lustre: Skipped 1 previous similar message [ 1237.251236] Lustre: lustre-MDT0001: new disk, initializing [ 1237.333348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1237.371424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1237.391963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1241.328408] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1249.328678] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1253.500737] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1258.061756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1258.131221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1258.138218] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1258.352096] Lustre: lustre-OST0000: new disk, initializing [ 1258.356750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1258.417919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.004247] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1267.245487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1267.253765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1269.167517] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1272.524835] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1272.691306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1276.426720] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1276.587587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1282.531650] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1282.534876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1282.541119] LustreError: Skipped 1 previous similar message [ 1282.542862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1282.553980] Lustre: Skipped 1 previous similar message [ 1282.569704] Lustre: Skipped 1 previous similar message [ 1287.653345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1287.660918] Lustre: Skipped 1 previous similar message [ 1292.255248] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1292.370600] Lustre: server umount lustre-OST0000 complete [ 1297.888024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.891383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1297.902954] Lustre: Skipped 1 previous similar message [ 1297.910821] Lustre: Skipped 1 previous similar message [ 1303.007940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1303.016049] Lustre: Skipped 1 previous similar message [ 1308.130656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1308.135744] Lustre: Skipped 1 previous similar message [ 1309.151321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1309.308040] Lustre: server umount lustre-MDT0000 complete [ 1312.803620] LustreError: 49138:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190271 with bad export cookie 18445689991721572752 [ 1312.809107] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1312.815576] LustreError: 49138:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1313.096991] Lustre: server umount lustre-MDT0001 complete [ 1321.591036] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 14:11:19 (1776190279) [ 1328.532470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1329.050637] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1329.113522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.939931] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1334.243695] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.361045] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.001707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.285244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1345.955357] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1352.963736] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1358.588642] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1365.948630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1366.278824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1370.340154] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1376.666238] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1386.707507] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-1414h12m25s [ 1388.014086] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1394.202322] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1396.367898] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1396.420843] LustreError: 56055:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1398.253606] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1398.261322] LNetError: 47354:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1398.269237] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1416.873992] Lustre: lustre-MDT0000: Client 101fc1a2-0818-42ac-8a18-77e0b95db328 (at 192.168.203.15@tcp) reconnecting [ 1436.480552] LustreError: 56055:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1438.216885] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1440.032898] Lustre: DEBUG MARKER: done [ 1444.324707] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1444.335855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.359304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1446.376844] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1446.391121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.320335] Lustre: server umount lustre-OST0000 complete [ 1455.073347] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1455.089119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.618984] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1456.627117] LNetError: 47351:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1456.639877] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1459.784852] Lustre: server umount lustre-MDT0000 complete [ 1461.743673] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.785718] LustreError: Skipped 1 previous similar message [ 1461.788705] LNetError: 120-3: Refusing connection from 192.168.203.115 for 127.0.0.2@tcp: No matching NI [ 1461.793979] LNetError: 47352:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1461.808384] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1463.873060] LustreError: 53677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190422 with bad export cookie 18445689991721573606 [ 1463.883406] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1463.890200] LustreError: 53677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1477.014979] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1481.170404] Key type lgssc unregistered [ 1481.587897] LNet: 58109:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.600353] LNet: Removed LNI 192.168.203.115@tcp [ 1482.827575] Key type .llcrypt unregistered [ 1482.831928] Key type ._llcrypt unregistered [ 1516.376855] alg: No test for adler32 (adler32-zlib) [ 1517.131653] Key type ._llcrypt registered [ 1517.144377] Key type .llcrypt registered [ 1517.268316] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 1530.584390] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1531.650764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1531.988144] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 1532.178202] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1532.185501] LNet: Accept secure, port 988 [ 1533.903134] Key type lgssc registered [ 1534.787097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1541.618113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1547.007328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1552.620915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1558.785030] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1564.781247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1564.844647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.050081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1566.083783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1566.181411] Lustre: lustre-MDT0000: new disk, initializing [ 1566.262810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.295812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1570.167480] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1581.293476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1581.387799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1581.511415] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1581.561634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1581.579424] Lustre: Skipped 1 previous similar message [ 1581.696820] Lustre: lustre-MDT0001: new disk, initializing [ 1581.816515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1581.858689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1581.870888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1587.182544] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1598.250706] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1603.050400] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1608.400937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1608.444843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1608.451554] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1608.630447] Lustre: lustre-OST0000: new disk, initializing [ 1608.634346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1608.708148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1612.088247] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1618.588201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1618.605986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1620.884976] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1625.734215] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1625.981651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1631.087359] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1631.438878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1634.272137] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1634.281753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.297977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1639.428424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1639.437603] Lustre: Skipped 1 previous similar message [ 1639.755620] Lustre: server umount lustre-OST0000 complete [ 1644.528250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.547295] Lustre: Skipped 1 previous similar message [ 1644.551399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.555324] Lustre: Skipped 2 previous similar messages [ 1649.632831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1654.751985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1654.751985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1654.751993] Lustre: Skipped 1 previous similar message [ 1658.335139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1658.438091] Lustre: server umount lustre-MDT0000 complete [ 1659.874715] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1659.891642] LustreError: Skipped 1 previous similar message [ 1660.902493] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1660.915265] LustreError: Skipped 1 previous similar message [ 1661.768529] LustreError: 62201:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190620 with bad export cookie 10108946566532272753 [ 1661.778875] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1661.786711] LustreError: 62201:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1662.074583] Lustre: server umount lustre-MDT0001 complete [ 1669.729882] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 14:17:07 (1776190627) [ 1670.874862] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1672.379655] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 14:17:10 (1776190630) [ 1682.206486] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1685.599960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.874686] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1685.932101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.721113] Lustre: Failing over lustre-MDT0000 [ 1686.730406] LustreError: 67223:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1686.738923] LustreError: 67223:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1686.744894] LustreError: 67223:0:(lod_dev.c:528:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1692.127533] Lustre: 67274:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776190645/real 1776190645] req@000000005408a2dd x1862470720312064/t0(0) o251->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776190651 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1692.236253] Lustre: server umount lustre-MDT0000 complete [ 1697.886794] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:17:36 (1776190656) [ 1703.831877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1704.271924] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1704.335891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.308411] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1709.537675] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1715.061223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.357021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1718.634064] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1724.525949] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1729.400533] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1734.341314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1734.543204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.442784] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1745.529319] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1750.530937] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1756.644589] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1756.650471] LustreError: Skipped 1 previous similar message [ 1756.653102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.661931] Lustre: Skipped 1 previous similar message [ 1756.665754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1761.252154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.264158] Lustre: Skipped 1 previous similar message [ 1764.013184] Lustre: server umount lustre-MDT0000 complete [ 1766.370649] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.388492] LustreError: Skipped 3 previous similar messages [ 1767.256281] LustreError: 67823:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190726 with bad export cookie 10108946566532273880 [ 1767.268774] LustreError: 67823:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1767.270789] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1770.520925] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1775.196409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1778.530532] Lustre: 60430:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776190730/real 1776190730] req@0000000042001ec2 x1862470720329856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776190737 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1778.562482] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1779.679136] Lustre: 60431:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776190731/real 1776190731] req@000000004254b62e x1862470720329920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776190738 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1783.307256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.775182] Lustre: 60429:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776190735/real 1776190735] req@000000006d906104 x1862470720330112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776190742 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1783.800552] Lustre: 60429:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1784.755835] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0x8c4a3157bf8f96d8 to 0x8c4a3157bf8f9e72 [ 1784.779204] Lustre: MGC192.168.203.115@tcp: Connection restored to 192.168.203.115@tcp (at 0@lo) [ 1785.070992] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.190873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1787.234505] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1789.777584] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1797.751963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.174810] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1802.715546] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1803.239231] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1810.271700] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1816.089771] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1823.720780] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.733952] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1823.740082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1823.756141] Lustre: Skipped 5 previous similar messages [ 1824.108981] Lustre: server umount lustre-MDT0000 complete [ 1824.114829] Lustre: Skipped 1 previous similar message [ 1827.422122] LustreError: 67822:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190786 with bad export cookie 10108946566532275826 [ 1827.426268] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.434880] LustreError: 67822:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1833.398453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1835.935132] Lustre: 60432:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776190787/real 1776190787] req@00000000a2e2ace5 x1862470720343936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776190794 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1841.697626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.197861] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0x8c4a3157bf8f9e72 to 0x8c4a3157bf8fa484 [ 1844.215572] Lustre: MGC192.168.203.115@tcp: Connection restored to (at 0@lo) [ 1844.402216] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1844.417417] LustreError: Skipped 5 previous similar messages [ 1844.506587] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1848.331981] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1855.737941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.975328] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1856.148528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1856.157706] Lustre: Skipped 2 previous similar messages [ 1856.218491] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1860.804339] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 1867.973508] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1873.265940] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1881.038417] Lustre: server umount lustre-MDT0000 complete [ 1881.042650] Lustre: Skipped 1 previous similar message [ 1881.569793] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1881.577378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1881.586819] Lustre: Skipped 3 previous similar messages [ 1883.647618] LustreError: 67822:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190842 with bad export cookie 10108946566532277380 [ 1883.655427] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1883.656278] LustreError: 67822:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1888.312645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1890.827205] Lustre: DEBUG MARKER: files compared the same [ 1898.467333] Lustre: 76965:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776190851/real 1776190851] req@000000006cb53500 x1862470720356736/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776190857 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1898.491285] Lustre: 76965:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1916.060510] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1918.728480] Key type lgssc unregistered [ 1919.015753] LNet: 77874:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.028498] LNet: Removed LNI 192.168.203.115@tcp [ 1919.745545] Key type .llcrypt unregistered [ 1919.748144] Key type ._llcrypt unregistered [ 1940.216578] alg: No test for adler32 (adler32-zlib) [ 1940.976295] Key type ._llcrypt registered [ 1940.981788] Key type .llcrypt registered [ 1941.113290] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1942.057231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1942.380491] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 1942.538372] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1942.551195] LNet: Accept secure, port 988 [ 1944.199140] Key type lgssc registered [ 1945.063551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1952.349230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.786476] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.826218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.931273] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug +malloc all 8 [ 1958.890452] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.001909] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.183690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.386889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1971.273210] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug +malloc all 8 [ 1978.091200] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1983.562393] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1989.880319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1990.133051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.638716] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug +malloc all 8 [ 1996.288388] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1999.767892] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2003.433724] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2011.616081] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2011.623511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.632515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2013.666846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2013.667668] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2013.668219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.679260] Lustre: Skipped 1 previous similar message [ 2016.672460] Lustre: server umount lustre-OST0000 complete [ 2022.880166] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2022.890098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.919981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2029.028286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2029.034694] Lustre: Skipped 3 previous similar messages [ 2034.144135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2034.145856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2034.184247] Lustre: Skipped 1 previous similar message [ 2034.351648] Lustre: server umount lustre-MDT0000 complete [ 2037.765893] LustreError: 79319:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190996 with bad export cookie 3550551820115614367 [ 2037.780832] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2037.781708] LustreError: 79319:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2038.112474] Lustre: server umount lustre-MDT0001 complete [ 2050.702335] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2053.219855] Key type lgssc unregistered [ 2053.481906] LNet: 82995:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.490161] LNet: Removed LNI 192.168.203.115@tcp [ 2054.123355] Key type .llcrypt unregistered [ 2054.126571] Key type ._llcrypt unregistered [ 2072.999592] alg: No test for adler32 (adler32-zlib) [ 2073.759126] Key type ._llcrypt registered [ 2073.760858] Key type .llcrypt registered [ 2073.883729] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2074.770797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2075.313490] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 2075.501493] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2075.505729] LNet: Accept secure, port 988 [ 2077.167274] Key type lgssc registered [ 2077.944418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2084.264721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2101.855767] LustreError: 84431:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 1: rc = -5 [ 2132.575229] LustreError: 84431:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 2162.271383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2166.121829] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2172.891248] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2178.780795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.864348] Lustre: *** cfs_fail_loc=706, val=0*** [ 2185.043914] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.103496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.155277] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 2188.134296] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2190.313695] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.191159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.479202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2194.543161] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 2198.286750] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2204.924951] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2209.410227] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2211.809936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.811664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2211.837574] Lustre: Skipped 1 previous similar message [ 2211.854859] Lustre: Skipped 1 previous similar message [ 2216.930860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2216.935576] Lustre: Skipped 1 previous similar message [ 2217.345310] Lustre: server umount lustre-OST0000 complete [ 2220.001613] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2220.008406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2220.022844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.048908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.049188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.175405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.188847] Lustre: Skipped 2 previous similar messages [ 2234.336027] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2234.510018] Lustre: server umount lustre-MDT0000 complete [ 2237.413390] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2237.440021] LustreError: Skipped 1 previous similar message [ 2239.307371] LustreError: 84439:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191198 with bad export cookie 2319174306484044260 [ 2239.315677] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.318996] LustreError: 84439:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2239.653769] Lustre: server umount lustre-MDT0001 complete [ 2252.623550] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2255.732606] Key type lgssc unregistered [ 2255.922060] LNet: 88172:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2255.927703] LNet: Removed LNI 192.168.203.115@tcp [ 2256.658719] Key type .llcrypt unregistered [ 2256.661673] Key type ._llcrypt unregistered [ 2278.442612] alg: No test for adler32 (adler32-zlib) [ 2279.196992] Key type ._llcrypt registered [ 2279.202384] Key type .llcrypt registered [ 2279.375635] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2280.597249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.999914] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 2281.214889] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2281.218166] LNet: Accept secure, port 988 [ 2282.912706] Key type lgssc registered [ 2284.130749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2292.520711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.228517] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2307.002234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2307.244474] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.326676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2312.415804] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2312.672613] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2312.699702] LustreError: Skipped 2 previous similar messages [ 2317.797628] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.830852] LustreError: Skipped 1 previous similar message [ 2320.602844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2320.866334] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.888261] LustreError: Skipped 2 previous similar messages [ 2320.888485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.962452] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 2325.253726] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2328.034296] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.042027] LustreError: Skipped 4 previous similar messages [ 2333.745462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.835360] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 2338.099654] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2353.632231] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2353.643686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.665800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2354.148244] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2354.170897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.191791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2354.198553] Lustre: Skipped 1 previous similar message [ 2356.126478] Lustre: server umount lustre-OST0000 complete [ 2364.390246] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2364.390579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.407284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2364.416648] Lustre: Skipped 1 previous similar message [ 2364.431349] Lustre: Skipped 1 previous similar message [ 2365.828953] Lustre: server umount lustre-MDT0000 complete [ 2369.616403] Lustre: server umount lustre-MDT0001 complete [ 2378.207218] Lustre: 93039:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776191331/real 1776191331] req@000000000d3ad317 x1862471505687552/t0(0) o251->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776191337 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2378.244249] Lustre: server umount MGS complete [ 2387.076465] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:29:05 (1776191345) [ 2412.606428] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 2420.086256] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2428.276913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2434.413799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2440.100524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2445.531862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.750436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.800178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.960423] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2462.596047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2462.649467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.851373] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2462.886187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2462.962142] Lustre: lustre-MDT0001: new disk, initializing [ 2463.058369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2463.082286] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2463.093078] LustreError: 97741:0:(lod_dev.c:528:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2467.068924] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2476.941781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2476.997735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2477.004411] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2477.234820] Lustre: lustre-OST0000: new disk, initializing [ 2477.241397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2477.318464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.185712] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2487.843534] Lustre: 99090:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000922ba4a0 x1862471505692352/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1776191458 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2490.088976] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2496.637993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2496.687475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2496.795651] Lustre: lustre-MDT0000: new disk, initializing [ 2496.972168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.996517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2498.016412] Lustre: 99593:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000bdc710d9 x1862471505695680/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1776191469 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2500.065284] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2501.018926] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2507.179942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2507.188123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2508.260870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2508.270696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2532.833352] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2532.852796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2532.892857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2540.009712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2540.023128] Lustre: Skipped 4 previous similar messages [ 2545.119336] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2545.308899] Lustre: server umount lustre-OST0000 complete [ 2550.242084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2550.246942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2550.257654] Lustre: Skipped 2 previous similar messages [ 2550.277271] Lustre: Skipped 3 previous similar messages [ 2555.282451] Lustre: server umount lustre-MDT0000 complete [ 2555.360076] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.376430] LustreError: Skipped 4 previous similar messages [ 2574.815162] Lustre: 100795:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776191527/real 1776191527] req@0000000083011acc x1862471505707520/t0(0) o251->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776191533 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2574.849477] Lustre: server umount MGS complete [ 2574.851097] Lustre: Skipped 1 previous similar message [ 2585.444736] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:32:23 (1776191543) [ 2602.940372] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2605.568214] Key type lgssc unregistered [ 2605.785851] LNet: 102203:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2605.796336] LNet: Removed LNI 192.168.203.115@tcp [ 2606.581916] Key type .llcrypt unregistered [ 2606.586434] Key type ._llcrypt unregistered [ 2621.584142] alg: No test for adler32 (adler32-zlib) [ 2622.339699] Key type ._llcrypt registered [ 2622.349536] Key type .llcrypt registered [ 2622.475403] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2623.253898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2623.549608] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 2623.742716] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2623.747888] LNet: Accept secure, port 988 [ 2625.391139] Key type lgssc registered [ 2626.250808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2632.731415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.119296] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.181843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.150056] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2639.331714] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.454196] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.946265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.309093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2649.264764] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2656.097089] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2661.850035] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2668.041524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2668.335869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2672.102635] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2679.843796] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2684.716942] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2695.136628] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2695.146599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.150064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2695.168558] Lustre: Skipped 1 previous similar message [ 2699.742892] Lustre: server umount lustre-OST0000 complete [ 2705.379248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.405683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2705.418832] Lustre: Skipped 2 previous similar messages [ 2710.198290] Lustre: server umount lustre-MDT0000 complete [ 2710.497822] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.518939] LustreError: Skipped 1 previous similar message [ 2713.951928] LustreError: 103450:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191672 with bad export cookie 16840864189901693035 [ 2713.965517] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2714.242683] Lustre: server umount lustre-MDT0001 complete [ 2725.942093] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2728.294777] Key type lgssc unregistered [ 2728.532796] LNet: 107146:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2728.544962] LNet: Removed LNI 192.168.203.115@tcp [ 2729.175258] Key type .llcrypt unregistered [ 2729.181872] Key type ._llcrypt unregistered [ 2742.645974] alg: No test for adler32 (adler32-zlib) [ 2743.407445] Key type ._llcrypt registered [ 2743.411336] Key type .llcrypt registered [ 2743.532106] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2744.617893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2744.920645] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 2745.094858] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2745.100318] LNet: Accept secure, port 988 [ 2746.759122] Key type lgssc registered [ 2747.659765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2752.878722] LustreError: 108273:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2752.895861] LustreError: 108278:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2752.899746] LustreError: 108273:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4982 [ 2752.908231] LustreError: 108273:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2752.923337] LustreError: 108273:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2752.931364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.317425] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.367054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2759.650293] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.503846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.764735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2765.675447] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2769.666326] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing lsmod [ 2771.982497] LustreError: 109585:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2771.995714] LustreError: 109586:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2772.001553] LustreError: 109585:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4988 [ 2772.008280] LustreError: 109585:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2772.016945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2772.018980] LustreError: 109585:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2772.279107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.353460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.356080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.366358] Lustre: Skipped 2 previous similar messages [ 2776.375392] Lustre: Skipped 2 previous similar messages [ 2780.370168] Lustre: server umount lustre-MDT0000 complete [ 2781.666919] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.674561] LustreError: Skipped 2 previous similar messages [ 2783.297739] LustreError: 108297:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191742 with bad export cookie 8845685647918350114 [ 2783.307079] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2783.521719] Lustre: server umount lustre-MDT0001 complete [ 2792.415409] Lustre: 110174:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776191745/real 1776191745] req@000000006a300e92 x1862471992222528/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776191751 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2792.758577] Lustre: server umount lustre-OST0000 complete [ 2801.145493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2801.421933] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.480110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.236244] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2806.768155] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.642363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.032252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2821.416099] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2828.302889] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2834.431077] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2840.028614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2840.255205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.883882] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2850.590662] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2853.619857] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2862.052254] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2862.065447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.087284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2864.103725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2864.109651] Lustre: Skipped 1 previous similar message [ 2867.177121] Lustre: server umount lustre-OST0000 complete [ 2873.314285] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2873.324570] LustreError: Skipped 1 previous similar message [ 2873.333563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2873.347124] Lustre: Skipped 1 previous similar message [ 2873.352653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.367535] Lustre: Skipped 1 previous similar message [ 2876.747101] Lustre: server umount lustre-MDT0000 complete [ 2879.461061] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.478946] LustreError: Skipped 2 previous similar messages [ 2880.046335] LustreError: 110632:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191839 with bad export cookie 8845685647918350793 [ 2880.060295] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2880.062149] LustreError: 110632:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2890.610961] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2893.023823] Key type lgssc unregistered [ 2893.291720] LNet: 114305:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2893.303643] LNet: Removed LNI 192.168.203.115@tcp [ 2893.992832] Key type .llcrypt unregistered [ 2893.995197] Key type ._llcrypt unregistered [ 2915.155390] alg: No test for adler32 (adler32-zlib) [ 2915.919433] Key type ._llcrypt registered [ 2915.924188] Key type .llcrypt registered [ 2916.065916] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2916.880427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2917.136239] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 2917.310527] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2917.315560] LNet: Accept secure, port 988 [ 2918.983128] Key type lgssc registered [ 2919.737332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2925.735532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.084797] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.150986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.909703] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2932.192913] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.312778] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.759187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.143572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2942.199235] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2949.732852] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2957.119927] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2963.391870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2963.897311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.493229] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 2974.992669] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2977.205962] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2983.445290] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2987.271841] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2992.614220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.620051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2992.626263] Lustre: Skipped 1 previous similar message [ 2992.640796] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2992.970504] Lustre: server umount lustre-OST0000 complete [ 2997.728769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.738970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2997.748816] Lustre: Skipped 1 previous similar message [ 2997.764776] Lustre: Skipped 2 previous similar messages [ 3002.757165] Lustre: server umount lustre-MDT0000 complete [ 3002.851916] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.866131] LustreError: Skipped 1 previous similar message [ 3005.842785] LustreError: 117721:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191964 with bad export cookie 9627345918534080102 [ 3005.844662] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3005.855535] LustreError: 117721:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3006.134250] Lustre: server umount lustre-MDT0001 complete [ 3016.394270] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3018.606362] Key type lgssc unregistered [ 3018.826619] LNet: 119529:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3018.837246] LNet: Removed LNI 192.168.203.115@tcp [ 3019.525729] Key type .llcrypt unregistered [ 3019.529557] Key type ._llcrypt unregistered [ 3033.014085] alg: No test for adler32 (adler32-zlib) [ 3033.773391] Key type ._llcrypt registered [ 3033.775873] Key type .llcrypt registered [ 3033.871910] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3034.817253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3035.188775] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 3035.386482] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3035.392327] LNet: Accept secure, port 988 [ 3037.080855] Key type lgssc registered [ 3037.923687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3046.818791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.420346] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3048.481513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.549187] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.912468] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3058.662958] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.140712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.541122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3068.030314] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3076.701945] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3085.068970] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3092.711057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3093.126597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.284894] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3099.646248] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3105.171517] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3113.960443] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3113.981226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.999288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3114.978884] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3114.980685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3115.001082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3115.038673] Lustre: Skipped 1 previous similar message [ 3118.459155] Lustre: server umount lustre-OST0000 complete [ 3125.217188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.228951] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3125.229684] Lustre: Skipped 1 previous similar message [ 3125.232064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3125.257645] Lustre: Skipped 1 previous similar message [ 3128.230485] Lustre: server umount lustre-MDT0000 complete [ 3130.338925] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.357579] LustreError: Skipped 1 previous similar message [ 3131.564650] LustreError: 120779:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192090 with bad export cookie 1730077143281155849 [ 3131.568672] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3131.572194] LustreError: 120779:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3131.867656] Lustre: server umount lustre-MDT0001 complete [ 3142.713303] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3145.062054] Key type lgssc unregistered [ 3145.246389] LNet: 124463:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3145.262314] LNet: Removed LNI 192.168.203.115@tcp [ 3145.873449] Key type .llcrypt unregistered [ 3145.878773] Key type ._llcrypt unregistered [ 3170.930406] alg: No test for adler32 (adler32-zlib) [ 3171.688658] Key type ._llcrypt registered [ 3171.696032] Key type .llcrypt registered [ 3171.863971] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3173.460127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3174.168059] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 3174.455356] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3174.467665] LNet: Accept secure, port 988 [ 3176.207474] Key type lgssc registered [ 3177.371898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3184.550443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.986552] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3186.027268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.926196] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3190.052649] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3195.362949] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.030849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.381868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3202.361665] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3209.493406] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3214.884527] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3220.856749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3221.218447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.946928] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3235.577977] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 3235.703483] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3241.926882] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 3245.307099] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3245.319221] Lustre: Skipped 2 previous similar messages [ 3249.158421] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 3249.165458] Lustre: Skipped 2 previous similar messages [ 3249.762895] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3249.832813] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3255.229921] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3255.237250] Lustre: Skipped 2 previous similar messages [ 3259.101906] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.15@tcp 192.168.0.[2,10]@tcp [ 3266.033818] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3266.038377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.065977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3266.550126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.456101] Lustre: server umount lustre-OST0000 complete [ 3271.650244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.672245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3271.675236] Lustre: Skipped 1 previous similar message [ 3276.768873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3276.787025] Lustre: Skipped 1 previous similar message [ 3277.906238] Lustre: server umount lustre-MDT0000 complete [ 3281.603155] LustreError: 125930:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192240 with bad export cookie 7734500060314798087 [ 3281.621479] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3281.947975] Lustre: server umount lustre-MDT0001 complete [ 3292.949390] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3295.392703] Key type lgssc unregistered [ 3295.600719] LNet: 130507:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3295.611200] LNet: Removed LNI 192.168.203.115@tcp [ 3296.337436] Key type .llcrypt unregistered [ 3296.340929] Key type ._llcrypt unregistered [ 3320.761808] alg: No test for adler32 (adler32-zlib) [ 3321.517341] Key type ._llcrypt registered [ 3321.520643] Key type .llcrypt registered [ 3321.663269] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3322.683862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3323.203411] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 3323.370931] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3323.374519] LNet: Accept secure, port 988 [ 3325.023154] Key type lgssc registered [ 3325.887611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3332.379079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.921876] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.961252] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3333.997863] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.203.15@tcp 192.168.0.[2,10]@tcp [ 3334.006844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3338.298182] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3339.236912] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3344.365718] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3346.980878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.327906] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3347.350768] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.15@tcp 192.168.0.[2,10]@tcp [ 3347.390049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3352.120505] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3360.193296] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3365.887768] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3372.465679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.784627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3377.914255] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3384.061772] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 3389.284487] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3399.649655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.655909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3399.676496] Lustre: Skipped 1 previous similar message [ 3404.773298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3404.779365] Lustre: Skipped 2 previous similar messages [ 3405.369267] Lustre: server umount lustre-OST0000 complete [ 3409.895900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.905660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3409.926038] Lustre: Skipped 1 previous similar message [ 3409.943252] Lustre: Skipped 1 previous similar message [ 3415.008545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3415.013328] Lustre: Skipped 1 previous similar message [ 3415.691157] Lustre: server umount lustre-MDT0000 complete [ 3419.555383] LustreError: 132094:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192378 with bad export cookie 15952371452272414959 [ 3419.562255] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3419.562786] LustreError: 132094:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3419.807378] Lustre: server umount lustre-MDT0001 complete [ 3428.101952] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3430.678939] Key type lgssc unregistered [ 3430.938313] LNet: 135822:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3430.957946] LNet: Removed LNI 192.168.203.115@tcp [ 3431.878539] Key type .llcrypt unregistered [ 3431.881964] Key type ._llcrypt unregistered [ 3465.435612] alg: No test for adler32 (adler32-zlib) [ 3466.198375] Key type ._llcrypt registered [ 3466.203501] Key type .llcrypt registered [ 3466.463813] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3480.892475] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3481.957051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3482.561548] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 3482.858800] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3482.872171] LNet: Accept secure, port 988 [ 3484.583120] Key type lgssc registered [ 3485.435638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3492.990453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3499.278808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3505.072775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3511.837381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3518.370320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3518.458088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3519.828934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3519.864492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3519.928635] Lustre: lustre-MDT0000: new disk, initializing [ 3519.986135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3520.005536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3524.721643] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3535.934147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3535.992833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3536.050525] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3536.122944] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3536.132443] Lustre: Skipped 1 previous similar message [ 3536.237177] Lustre: lustre-MDT0001: new disk, initializing [ 3536.422906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3536.477624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3536.489692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3541.288320] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3550.078362] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3555.649703] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3562.107966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3562.186967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3562.206871] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3562.530588] Lustre: lustre-OST0000: new disk, initializing [ 3562.534441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3562.652856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3568.423654] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3569.792563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3569.803610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3578.949949] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3583.971595] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3584.303580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3589.851111] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3590.327525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3595.744895] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3595.750972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.768178] LustreError: Skipped 1 previous similar message [ 3595.768857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3595.792336] Lustre: Skipped 1 previous similar message [ 3599.187269] Lustre: server umount lustre-OST0000 complete [ 3606.001156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.024693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3606.037390] Lustre: Skipped 2 previous similar messages [ 3609.742640] Lustre: server umount lustre-MDT0000 complete [ 3611.106045] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3611.125746] LustreError: Skipped 1 previous similar message [ 3612.131346] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3612.144568] LustreError: Skipped 1 previous similar message [ 3614.517742] LustreError: 139377:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192573 with bad export cookie 4293184006183702760 [ 3614.526729] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3614.534773] LustreError: 139377:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3614.887404] Lustre: server umount lustre-MDT0001 complete [ 3624.140834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3624.725385] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3624.810430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.489628] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3630.054595] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3635.174171] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3638.180499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3638.585352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3643.055958] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3654.173462] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3663.858743] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3670.634306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3670.991049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3675.322437] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3682.300982] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3686.988619] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3687.227797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3691.422363] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3691.722996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3701.985526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3702.100543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3702.110765] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3702.312645] Lustre: lustre-OST0001: new disk, initializing [ 3702.325695] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3702.464136] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3707.546378] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3712.136482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3712.149706] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3716.537082] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3716.740825] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3721.080989] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3721.462627] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3727.739483] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b8844889000.ost_server_uuid 40 [ 3729.407871] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b8844889000.ost_server_uuid in FULL state after 0 sec [ 3740.399479] Lustre: server umount lustre-OST0001 complete [ 3743.202037] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3743.218779] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.223558] LustreError: Skipped 1 previous similar message [ 3743.233850] Lustre: Skipped 1 previous similar message [ 3743.242217] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.270642] LustreError: Skipped 1 previous similar message [ 3746.797153] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3746.811452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3746.834298] Lustre: Skipped 1 previous similar message [ 3746.838957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.453569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.459227] Lustre: Skipped 3 previous similar messages [ 3758.585866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3758.592792] Lustre: Skipped 1 previous similar message [ 3759.072422] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3759.250566] Lustre: server umount lustre-OST0000 complete [ 3763.690110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.733840] Lustre: Skipped 2 previous similar messages [ 3768.804387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3768.823702] Lustre: Skipped 3 previous similar messages [ 3768.995473] Lustre: server umount lustre-MDT0000 complete [ 3773.492964] LustreError: 144528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192732 with bad export cookie 4293184006183703614 [ 3773.494628] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3773.522324] LustreError: 144528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3795.845502] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3798.430810] Key type lgssc unregistered [ 3798.732809] LNet: 150397:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3798.743537] LNet: Removed LNI 192.168.203.115@tcp [ 3799.514858] Key type .llcrypt unregistered [ 3799.517199] Key type ._llcrypt unregistered [ 3832.263209] alg: No test for adler32 (adler32-zlib) [ 3833.018532] Key type ._llcrypt registered [ 3833.020525] Key type .llcrypt registered [ 3833.106300] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3845.190346] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3846.216196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3846.608273] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 3846.813777] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3846.820084] LNet: Accept secure, port 988 [ 3848.656015] Key type lgssc registered [ 3849.599558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3856.901487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3865.704915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3873.898530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3882.045754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3890.302213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3890.393744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.558492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3891.609080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3891.768748] Lustre: lustre-MDT0000: new disk, initializing [ 3891.952038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3892.021582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3896.656527] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3907.731327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3907.784157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.850775] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3907.922262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3907.929882] Lustre: Skipped 1 previous similar message [ 3908.019333] Lustre: lustre-MDT0001: new disk, initializing [ 3908.113902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3908.150072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3908.166720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3912.511720] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3922.053547] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3928.405628] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3934.758234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3934.818259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3934.824452] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3934.988287] Lustre: lustre-OST0000: new disk, initializing [ 3934.991559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3935.058621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3939.116963] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 3940.438173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3940.445451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3948.723328] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3953.354175] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3953.606718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3958.533992] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3958.803370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3961.318325] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3961.322043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.324690] LustreError: Skipped 1 previous similar message [ 3961.325332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3961.338524] Lustre: Skipped 1 previous similar message [ 3961.353900] Lustre: Skipped 1 previous similar message [ 3966.445713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3966.454501] Lustre: Skipped 1 previous similar message [ 3971.551909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3971.564585] Lustre: Skipped 1 previous similar message [ 3975.135178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3975.356085] Lustre: server umount lustre-OST0000 complete [ 3979.746066] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3979.757430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3979.784940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3981.794043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.908863] Lustre: server umount lustre-MDT0000 complete [ 3986.918205] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3986.932857] LustreError: Skipped 1 previous similar message [ 3990.459214] LustreError: 154491:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192949 with bad export cookie 14425911677838148154 [ 3990.466119] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3990.468591] LustreError: 154491:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3990.822384] Lustre: server umount lustre-MDT0001 complete [ 3999.685940] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 14:55:57 (1776192957) [ 4017.585611] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 4027.752126] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4038.095810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4047.156913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4056.747251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4064.674934] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4072.790699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4072.922175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.289828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4073.354816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4073.617970] Lustre: lustre-MDT0000: new disk, initializing [ 4073.796379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4073.835612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4078.979447] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4091.366834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4091.446718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4091.585494] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4091.619696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4091.627558] Lustre: Skipped 1 previous similar message [ 4091.759789] Lustre: lustre-MDT0001: new disk, initializing [ 4091.857152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4091.920923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4091.933701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4096.160274] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4105.311311] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4111.039736] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4117.422293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4117.485737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4117.499901] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4117.721678] Lustre: lustre-OST0000: new disk, initializing [ 4117.736127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4117.828940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4122.329327] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4123.765080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4123.785788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4132.936847] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4137.397162] Lustre: Failing over lustre-OST0000 [ 4137.597680] Lustre: server umount lustre-OST0000 complete [ 4145.632322] Lustre: 152718:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193097/real 1776193097] req@00000000c566abd9 x1862473146731136/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1776193104 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4145.647337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.655725] Lustre: 152716:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193098/real 1776193098] req@0000000074ed1d1d x1862473146731264/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776193105 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4151.776163] Lustre: 152717:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193103/real 1776193103] req@00000000b74bbed8 x1862473146732096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776193110 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4151.827958] Lustre: 152717:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4153.830623] Lustre: 152717:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193104/real 1776193104] req@00000000409e9b5d x1862473146732544/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776193111 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4153.870160] Lustre: 152717:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4159.116353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4159.360271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4159.389705] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4162.409947] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4164.272355] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4164.272785] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 4164.313698] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4165.307701] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4170.340526] Lustre: Failing over lustre-MDT0000 [ 4170.757404] Lustre: server umount lustre-MDT0000 complete [ 4172.780716] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.788675] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4172.807554] Lustre: Skipped 3 previous similar messages [ 4172.840455] LustreError: Skipped 2 previous similar messages [ 4174.497085] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4177.890490] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4177.915595] LustreError: Skipped 2 previous similar messages [ 4179.935865] Lustre: 152716:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193131/real 1776193131] req@00000000e8015143 x1862473146739968/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776193138 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4179.973682] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4185.078365] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4185.103641] LustreError: Skipped 7 previous similar messages [ 4193.087841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4195.301719] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.337165] LustreError: Skipped 9 previous similar messages [ 4196.351091] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0xc833223d3eb3c590 to 0xc833223d3eb3cb39 [ 4196.388742] Lustre: MGC192.168.203.115@tcp: Connection restored to (at 0@lo) [ 4196.412782] Lustre: Skipped 1 previous similar message [ 4196.756377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.837314] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4196.988780] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4201.446541] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4201.968544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4201.987029] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4202.024273] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 4210.656739] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4210.678635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.721225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4210.737980] Lustre: Skipped 2 previous similar messages [ 4216.310251] Lustre: server umount lustre-OST0000 complete [ 4222.432204] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4222.442798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4222.459268] Lustre: Skipped 3 previous similar messages [ 4227.553673] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.576704] LustreError: Skipped 4 previous similar messages [ 4231.038678] LustreError: 161483:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193190 with bad export cookie 14425911677838150457 [ 4231.042935] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4231.049416] LustreError: 161483:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4244.164416] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 4247.242650] Key type lgssc unregistered [ 4247.486852] LNet: 167350:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4247.498967] LNet: Removed LNI 192.168.203.115@tcp [ 4248.193426] Key type .llcrypt unregistered [ 4248.196253] Key type ._llcrypt unregistered [ 4269.997253] alg: No test for adler32 (adler32-zlib) [ 4270.751815] Key type ._llcrypt registered [ 4270.753732] Key type .llcrypt registered [ 4270.885443] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4271.889498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4272.259288] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 4272.458798] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 4272.463716] LNet: Accept secure, port 988 [ 4274.135247] Key type lgssc registered [ 4275.449283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4283.306573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4284.866423] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4284.918801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4289.682327] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4290.033557] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4295.137956] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4297.969882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4298.339856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4303.556309] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4312.092903] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4318.708235] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4327.364076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.786159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4332.020157] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4335.994374] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 4340.415116] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4436.640556] Lustre: lustre-MDT0000: Client 004e4421-d226-4caf-8cf7-174e0a35dbbe (at 192.168.203.15@tcp) reconnecting [ 4502.200192] Lustre: lustre-MDT0000: Client 004e4421-d226-4caf-8cf7-174e0a35dbbe (at 192.168.203.15@tcp) reconnecting [ 4563.621975] Lustre: lustre-MDT0000: Client 004e4421-d226-4caf-8cf7-174e0a35dbbe (at 192.168.203.15@tcp) reconnecting [ 4620.993686] Lustre: lustre-MDT0000: Client 004e4421-d226-4caf-8cf7-174e0a35dbbe (at 192.168.203.15@tcp) reconnecting [ 4747.961482] Lustre: lustre-MDT0000: Client 004e4421-d226-4caf-8cf7-174e0a35dbbe (at 192.168.203.15@tcp) reconnecting [ 4892.514022] Lustre: Failing over lustre-MDT0000 [ 4892.755822] Lustre: server umount lustre-MDT0000 complete [ 4894.179531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4894.187169] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4894.192180] Lustre: Skipped 2 previous similar messages [ 4894.214408] LustreError: Skipped 2 previous similar messages [ 4899.301663] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4899.318967] LustreError: Skipped 3 previous similar messages [ 4901.345666] Lustre: 168125:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193853/real 1776193853] req@0000000044c80d98 x1862473593777344/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776193860 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4901.366787] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4907.507916] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4907.526956] LustreError: Skipped 10 previous similar messages [ 4913.229977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4918.772384] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0xe7e8998f51226891 to 0xe7e8998f5124d4b9 [ 4918.824482] Lustre: MGC192.168.203.115@tcp: Connection restored to 192.168.203.115@tcp (at 0@lo) [ 4918.909231] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (not set up) [ 4919.187430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4919.251936] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4919.941034] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4923.661850] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 4924.405493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4924.430726] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4924.481530] Lustre: lustre-OST0000: deleting orphan objects from 0x0:100 to 0x0:129 [ 4933.202671] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4934.855688] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4955.108163] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4955.110168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4955.123682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4955.135489] Lustre: Skipped 2 previous similar messages [ 4955.138432] Lustre: Skipped 1 previous similar message [ 4960.225377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4960.235532] Lustre: Skipped 3 previous similar messages [ 4965.343118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4965.345523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4965.358244] Lustre: Skipped 2 previous similar messages [ 4965.446286] Lustre: server umount lustre-MDT0000 complete [ 4968.065768] LustreError: 168793:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193927 with bad export cookie 16710775258178966713 [ 4968.068306] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4968.079225] LustreError: 168793:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4970.463892] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4970.464511] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4970.483580] LustreError: Skipped 12 previous similar messages [ 4970.504863] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4974.457747] Lustre: server umount lustre-MDT0001 complete [ 4984.287086] Lustre: server umount lustre-OST0000 complete [ 4993.370734] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 5001.942329] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5011.799101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5018.099565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5023.804621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5029.214039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5035.638964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5035.767212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5036.041211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5036.080481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5036.182639] Lustre: lustre-MDT0000: new disk, initializing [ 5036.315083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5036.359796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5041.434224] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5051.956506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5052.020317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.112980] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5052.158723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5052.169786] Lustre: Skipped 1 previous similar message [ 5052.270351] Lustre: lustre-MDT0001: new disk, initializing [ 5052.370150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5052.417279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5052.430198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5057.195534] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5066.193782] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5071.947000] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5078.919878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5078.990676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5078.997339] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5079.357060] Lustre: lustre-OST0000: new disk, initializing [ 5079.372454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5079.518687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5084.202866] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5084.704862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5084.717653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5095.324729] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5100.590404] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5100.835558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5107.194333] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5107.543585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5110.759279] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5110.765695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5110.775270] LustreError: Skipped 1 previous similar message [ 5110.799776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5110.812754] Lustre: Skipped 2 previous similar messages [ 5115.831122] Lustre: server umount lustre-OST0000 complete [ 5120.996986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5121.010854] Lustre: Skipped 2 previous similar messages [ 5125.545382] Lustre: server umount lustre-MDT0000 complete [ 5126.112060] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5126.130551] LustreError: Skipped 2 previous similar messages [ 5129.517914] LustreError: 177846:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194088 with bad export cookie 16710775258178967434 [ 5129.523298] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5129.526135] LustreError: 177846:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5138.241672] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:14:56 (1776194096) [ 5156.146682] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 5165.546771] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5176.322524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5181.905935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5187.563586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5192.590377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5199.505178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5199.563172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.772255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5199.812663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5199.910116] Lustre: lustre-MDT0000: new disk, initializing [ 5199.987900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5200.014226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5204.147887] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5214.202890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5214.318188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.465785] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5214.515281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5214.519820] Lustre: Skipped 1 previous similar message [ 5214.600774] Lustre: lustre-MDT0001: new disk, initializing [ 5214.754699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5214.763588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5219.089349] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5229.153151] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5235.559320] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5241.611388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5241.687108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5241.700441] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5241.896855] Lustre: lustre-OST0000: new disk, initializing [ 5241.900233] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5246.538551] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5251.143780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5251.150688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5255.022511] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5261.792049] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5261.798133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5261.799071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5261.805230] Lustre: Skipped 1 previous similar message [ 5261.815052] Lustre: Skipped 3 previous similar messages [ 5267.950982] Lustre: server umount lustre-OST0000 complete [ 5267.953972] Lustre: Skipped 1 previous similar message [ 5281.212777] LustreError: 182744:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194240 with bad export cookie 16710775258178968288 [ 5281.221834] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5281.226066] LustreError: 182744:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5288.956324] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:17:27 (1776194247) [ 5302.732704] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 5310.366528] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5319.286519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5325.248916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5331.900602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5339.155602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5346.451326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5346.547982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5346.920966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5346.992135] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5347.137517] Lustre: lustre-MDT0000: new disk, initializing [ 5347.292866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5347.300464] Lustre: Skipped 2 previous similar messages [ 5347.360347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5352.484178] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5363.497821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5363.579471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.709815] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5364.005505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5369.174369] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5378.858905] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5384.655441] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5391.552593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5391.654952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5391.665682] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5391.909678] Lustre: lustre-OST0000: new disk, initializing [ 5391.913342] Lustre: Skipped 1 previous similar message [ 5391.921256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5391.931357] Lustre: Skipped 2 previous similar messages [ 5396.688858] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5398.091130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5398.107506] Lustre: Skipped 1 previous similar message [ 5398.112184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5406.533936] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5412.837210] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5412.844819] LustreError: Skipped 1 previous similar message [ 5412.848843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5412.858622] Lustre: Skipped 2 previous similar messages [ 5412.864963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5412.869027] Lustre: Skipped 6 previous similar messages [ 5415.069393] Lustre: server umount lustre-OST0000 complete [ 5415.076162] Lustre: Skipped 2 previous similar messages [ 5429.216396] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5429.249960] LustreError: Skipped 3 previous similar messages [ 5429.693526] LustreError: 189211:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194388 with bad export cookie 16710775258178969373 [ 5429.709271] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5429.711785] LustreError: 189211:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5442.563288] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5445.340154] Key type lgssc unregistered [ 5445.689996] LNet: 193177:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5445.703857] LNet: Removed LNI 192.168.203.115@tcp [ 5446.518458] Key type .llcrypt unregistered [ 5446.521648] Key type ._llcrypt unregistered [ 5468.058857] alg: No test for adler32 (adler32-zlib) [ 5468.813613] Key type ._llcrypt registered [ 5468.816329] Key type .llcrypt registered [ 5468.930992] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5470.072357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5470.949186] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 5471.192992] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5471.206810] LNet: Accept secure, port 988 [ 5472.975204] Key type lgssc registered [ 5474.429584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5482.733337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5484.133938] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5484.216707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5488.647102] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5489.639312] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5494.757483] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5496.933586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5497.321374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5501.267674] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5509.113402] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5515.250100] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5522.499550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5522.835873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5526.461358] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5533.901289] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5534.119683] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 5553.639565] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5553.655782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5553.671611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5554.655958] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5554.670390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5554.671320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5554.688034] Lustre: Skipped 1 previous similar message [ 5556.946083] Lustre: server umount lustre-OST0000 complete [ 5563.872039] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5563.882885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5563.905453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5566.413869] Lustre: server umount lustre-MDT0000 complete [ 5569.770697] LustreError: 196594:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194528 with bad export cookie 8627510007979798008 [ 5569.779715] LustreError: 196594:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5569.780079] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5570.073269] Lustre: server umount lustre-MDT0001 complete [ 5581.557245] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5584.138258] Key type lgssc unregistered [ 5584.447470] LNet: 198321:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5584.462328] LNet: Removed LNI 192.168.203.115@tcp [ 5585.280560] Key type .llcrypt unregistered [ 5585.282565] Key type ._llcrypt unregistered [ 5605.794246] alg: No test for adler32 (adler32-zlib) [ 5606.550374] Key type ._llcrypt registered [ 5606.562040] Key type .llcrypt registered [ 5606.700929] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5607.596826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5608.063703] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 5608.250486] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5608.255960] LNet: Accept secure, port 988 [ 5609.991127] Key type lgssc registered [ 5611.069153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5617.806371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5619.322910] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5619.408437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5623.369174] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5624.805986] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.922857] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5632.271711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.586793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5636.925699] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5644.750329] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5650.708537] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5656.790432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5657.154693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5661.495524] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5668.621544] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5669.426360] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5679.586137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5679.589670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5679.602441] Lustre: Skipped 1 previous similar message [ 5679.614949] Lustre: Skipped 1 previous similar message [ 5684.518456] Lustre: server umount lustre-OST0000 complete [ 5692.845236] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9b8845a2f800.ost_server_uuid 40 [ 5694.621443] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5709.280843] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5709.294167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.306406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5709.309721] Lustre: Skipped 1 previous similar message [ 5711.342737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.344732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5711.367887] Lustre: Skipped 1 previous similar message [ 5715.402753] Lustre: server umount lustre-MDT0000 complete [ 5717.472478] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5717.481871] LustreError: Skipped 1 previous similar message [ 5719.097762] LustreError: 199762:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194678 with bad export cookie 5775064585350233913 [ 5719.108147] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5719.116360] LustreError: 199762:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5719.510863] Lustre: server umount lustre-MDT0001 complete [ 5728.243486] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:24:46 (1776194686) [ 5737.115153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5737.616576] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5737.718648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5743.494860] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5748.204847] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5748.228306] LustreError: Skipped 1 previous similar message [ 5754.116893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5754.703090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5759.119192] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5767.039550] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5773.401280] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5781.238264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5781.463352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.626090] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5789.694309] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 5793.365475] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5799.756491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5799.831639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5799.841545] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5799.934445] Lustre: lustre-OST0001: new disk, initializing [ 5799.939733] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5800.001255] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5804.607883] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5808.231415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5808.238806] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5815.096641] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5818.849829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5818.869300] Lustre: Skipped 1 previous similar message [ 5818.869541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5818.877709] Lustre: Skipped 3 previous similar messages [ 5823.987030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5824.015404] Lustre: Skipped 2 previous similar messages [ 5824.319850] Lustre: server umount lustre-OST0000 complete [ 5826.187498] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5826.212726] LustreError: Skipped 1 previous similar message [ 5832.180713] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5832.536124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5838.466952] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5838.680649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5854.698599] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5854.714552] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.746205] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5854.765939] Lustre: Skipped 1 previous similar message [ 5858.584961] Lustre: server umount lustre-OST0001 complete [ 5864.939045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.953404] Lustre: Skipped 2 previous similar messages [ 5871.074566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5871.085557] Lustre: Skipped 5 previous similar messages [ 5877.215244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5877.421680] Lustre: server umount lustre-MDT0000 complete [ 5881.108199] LustreError: 203753:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194840 with bad export cookie 5775064585350234977 [ 5881.114112] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5881.123385] LustreError: 203753:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5887.161759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5890.703232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5894.277132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5897.631502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5905.018845] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:27:42 (1776194862) [ 5911.469744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.813847] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5911.860729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5912.102991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5916.850930] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5926.079237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.202745] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5926.254320] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5932.197712] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5940.220260] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5946.764872] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5954.123777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5954.325176] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5958.982881] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5963.399324] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 5966.868956] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5973.561414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5973.638133] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5978.349823] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 5982.385691] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 5986.564437] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5989.350317] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5989.367276] LustreError: Skipped 1 previous similar message [ 5989.372932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5989.394600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5989.400631] Lustre: Skipped 2 previous similar messages [ 5995.185402] Lustre: server umount lustre-OST0000 complete [ 5995.188028] Lustre: Skipped 1 previous similar message [ 5997.696923] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5997.708137] LustreError: Skipped 16 previous similar messages [ 6013.410366] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6013.430643] LustreError: Skipped 1 previous similar message [ 6023.648903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.663189] Lustre: Skipped 4 previous similar messages [ 6030.832894] LustreError: 211004:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194989 with bad export cookie 5775064585350236363 [ 6030.837430] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6030.842451] LustreError: 211004:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6031.138228] Lustre: server umount lustre-MDT0001 complete [ 6031.151879] Lustre: Skipped 2 previous similar messages [ 6038.117962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6042.648428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6047.465173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6051.242780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6058.586382] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:30:16 (1776195016) [ 6076.293502] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 6084.786898] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6093.655511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6099.817315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6105.253939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6110.805404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6116.631925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6116.698037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.907229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6116.953641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6117.050399] Lustre: lustre-MDT0000: new disk, initializing [ 6117.123971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6117.127059] Lustre: Skipped 3 previous similar messages [ 6117.137968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6121.100800] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6131.813372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6131.849823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6131.905826] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6131.934893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6131.938469] Lustre: Skipped 1 previous similar message [ 6131.996210] Lustre: lustre-MDT0001: new disk, initializing [ 6132.091494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6132.102535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6137.873885] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6148.371961] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6155.539316] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6164.300357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6164.348344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6164.354666] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6164.653300] Lustre: lustre-OST0000: new disk, initializing [ 6164.661520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6170.247632] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6170.692853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6170.701159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6180.129224] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6185.148689] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6185.380689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6189.985295] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6190.161481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6196.709553] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6196.720101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.722084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6196.722092] Lustre: Skipped 7 previous similar messages [ 6196.724184] LustreError: Skipped 1 previous similar message [ 6196.772169] Lustre: Skipped 1 previous similar message [ 6198.472586] Lustre: server umount lustre-OST0000 complete [ 6212.065490] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6212.087028] LustreError: Skipped 15 previous similar messages [ 6212.483994] LustreError: 218435:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195171 with bad export cookie 5775064585350238099 [ 6212.495103] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6212.502061] LustreError: 218435:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6221.018565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6225.493806] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6232.516055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.105656] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6246.438538] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6253.301612] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6260.999963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6261.381833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6261.388318] Lustre: Skipped 4 previous similar messages [ 6265.746564] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6272.404939] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6276.384437] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6276.576612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6280.382100] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6280.550185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6282.208139] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6282.219959] LustreError: Skipped 1 previous similar message [ 6293.904279] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6298.034266] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6327.103879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6329.580597] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.607798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.607841] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 6329.614393] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 6329.634906] Lustre: Skipped 1 previous similar message [ 6329.644368] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 6332.776723] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6340.498984] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6342.310628] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6350.305446] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6350.309735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6350.310673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6350.310678] Lustre: Skipped 8 previous similar messages [ 6350.334492] Lustre: Skipped 5 previous similar messages [ 6353.199663] Lustre: server umount lustre-OST0000 complete [ 6353.203127] Lustre: Skipped 3 previous similar messages [ 6365.669959] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6365.701175] LustreError: Skipped 4 previous similar messages [ 6365.998183] LustreError: 222779:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195324 with bad export cookie 5775064585350238953 [ 6366.005491] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6366.011465] LustreError: 222779:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6374.088740] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:35:32 (1776195332) [ 6379.410504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6383.479559] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6391.320743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6394.969686] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6401.769681] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6407.060735] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6413.048907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6417.242746] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6424.979227] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6429.165171] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6431.448594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6436.440881] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6436.716518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6444.282343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6444.394957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6444.401092] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6444.539417] Lustre: lustre-OST0001: new disk, initializing [ 6444.543390] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6448.407698] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6453.322394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6453.332594] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6458.102578] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6462.631898] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6462.825913] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6466.926634] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6467.122932] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6469.091220] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6469.101049] LustreError: Skipped 2 previous similar messages [ 6481.562804] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6481.901254] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6487.365781] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6487.541562] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6515.863841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6520.158497] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6521.323688] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6521.337025] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6521.341423] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 6521.355028] Lustre: Skipped 1 previous similar message [ 6527.473724] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6529.229536] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6565.856503] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6565.874197] LustreError: Skipped 2 previous similar messages [ 6575.808516] LustreError: 235056:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195534 with bad export cookie 5775064585350240367 [ 6575.817712] LustreError: 235056:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6575.818296] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6588.132651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6591.797307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6595.496663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6599.750575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6606.793309] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:39:24 (1776195564) [ 6613.545563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6613.839556] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6613.869945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6614.057054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6614.061717] Lustre: Skipped 6 previous similar messages [ 6618.362584] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6626.773237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6626.896046] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6626.939687] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6631.217224] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6638.445430] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6644.811323] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6650.682298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6650.802874] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6654.916677] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6662.037118] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6670.715912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6670.842937] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6675.938844] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6684.688368] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6686.387637] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6691.837273] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6692.035988] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6697.100376] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6697.297150] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6703.023213] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b88492da800.ost_server_uuid 40 [ 6704.684526] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b88492da800.ost_server_uuid in IDLE state after 0 sec [ 6705.900880] Lustre: Permanently deactivating lustre-OST0001 [ 6705.936705] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6710.496731] Lustre: Permanently reactivating lustre-OST0001 [ 6715.883974] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6715.904274] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6715.910570] Lustre: Skipped 10 previous similar messages [ 6715.922617] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6715.925388] Lustre: Skipped 17 previous similar messages [ 6729.183578] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6729.278337] Lustre: server umount lustre-OST0001 complete [ 6729.281245] Lustre: Skipped 7 previous similar messages [ 6733.285120] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6733.290919] LustreError: Skipped 40 previous similar messages [ 6747.103395] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6760.166774] LustreError: 237916:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195719 with bad export cookie 5775064585350241641 [ 6760.177849] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6760.178180] LustreError: 237916:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6765.368427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6769.068085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6772.713595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6776.907614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6783.582964] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:42:21 (1776195741) [ 6784.776314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6791.915201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6792.324649] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6792.389804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6792.393882] Lustre: Skipped 5 previous similar messages [ 6797.427969] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6806.717113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6806.838345] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6811.050911] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6819.858806] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6826.666444] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6834.279439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6834.373809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6834.659683] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6834.698649] Lustre: Permanently deactivating lustre-OST0000 [ 6839.793758] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6841.904308] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6841.965227] LustreError: 247162:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6848.155630] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6855.141702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6855.265516] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6859.667345] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6862.531539] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 6867.322336] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6868.883504] Lustre: Permanently reactivating lustre-OST0000 [ 6868.905813] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 6868.911257] Lustre: Skipped 4 previous similar messages [ 6872.073883] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6872.113065] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6872.122854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6872.138685] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 6872.147758] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 6909.407140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6922.582357] LustreError: 244762:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195881 with bad export cookie 5775064585350243251 [ 6922.584381] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6922.590060] LustreError: 244762:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6933.325969] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6935.784397] Key type lgssc unregistered [ 6936.034215] LNet: 249734:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6936.054551] LNet: Removed LNI 192.168.203.115@tcp [ 6936.731815] Key type .llcrypt unregistered [ 6936.735419] Key type ._llcrypt unregistered [ 6957.702383] alg: No test for adler32 (adler32-zlib) [ 6958.463500] Key type ._llcrypt registered [ 6958.465671] Key type .llcrypt registered [ 6958.641123] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6959.541415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6959.954444] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 6960.176536] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6960.190364] LNet: Accept secure, port 988 [ 6962.071169] Key type lgssc registered [ 6963.878918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6968.674895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6976.203821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6977.631242] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6977.729656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6982.405677] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 6983.143211] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.257903] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6990.703815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6990.757731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6990.879995] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6990.918940] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6990.931283] Lustre: Permanently deactivating lustre-MDT0001 [ 6991.253763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6995.709359] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7000.094774] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7003.052512] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7008.561589] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7014.256838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7014.564449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7018.596959] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7021.729703] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7021.754952] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:163 to 0x280000400:193 [ 7026.063303] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7027.171231] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.214661] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 7032.415269] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7032.600304] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7037.686877] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7037.945220] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:129 [ 7037.973814] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 [ 7046.932323] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7050.165772] Lustre: Permanently reactivating lustre-MDT0001 [ 7050.187443] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7050.195435] Lustre: Skipped 3 previous similar messages [ 7056.030181] Lustre: lustre-MDT0001: Client e258cb42-3006-4b81-a1a7-d56e8d2146e5 (at 192.168.203.15@tcp) reconnecting [ 7060.030845] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.046344] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7060.062949] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7060.080369] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 7062.498273] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7064.368340] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7070.791777] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 7070.996946] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7075.519434] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 7075.710893] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7078.361777] Lustre: Permanently deactivating lustre-MDT0001 [ 7078.382943] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7078.390928] Lustre: Skipped 2 previous similar messages [ 7085.668961] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7098.852980] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7098.871570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.894992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7104.486156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7104.491845] Lustre: Skipped 7 previous similar messages [ 7104.773083] Lustre: server umount lustre-MDT0000 complete [ 7108.212141] LustreError: 251232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196067 with bad export cookie 4906328747019435833 [ 7108.226655] LustreError: 251232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7108.239908] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7108.536848] Lustre: server umount lustre-MDT0001 complete [ 7114.296412] Lustre: server umount lustre-OST0000 complete [ 7125.747329] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:48:03 (1776196083) [ 7136.631378] Lustre: server umount lustre-OST0001 complete [ 7142.917955] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 7151.903851] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7161.842473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7167.976132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7173.618313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7179.752329] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7186.056651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7186.149966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7186.373161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7186.377592] Lustre: Skipped 2 previous similar messages [ 7186.399468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7186.503304] Lustre: lustre-MDT0000: new disk, initializing [ 7186.590809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7186.618758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7190.704446] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7200.766711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7200.841852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.894516] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7200.924447] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7200.929262] Lustre: Skipped 1 previous similar message [ 7200.998724] Lustre: lustre-MDT0001: new disk, initializing [ 7201.105446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7201.155060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7201.176186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7206.252303] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7215.682966] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7221.317937] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7227.480782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7227.573226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7227.581414] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7227.774271] Lustre: lustre-OST0000: new disk, initializing [ 7227.778386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7227.856656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7232.193674] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7235.630712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7235.639797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7241.607419] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7243.879871] LustreError: 259762:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7252.473291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7252.569476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7252.588912] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7252.732792] Lustre: lustre-OST0001: new disk, initializing [ 7252.736240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7253.883116] LustreError: 259762:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 7253.911614] LustreError: 259762:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7256.973185] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7262.352639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7262.368326] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7263.976182] LustreError: 259762:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 7265.748761] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7267.441183] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7271.078180] Lustre: lustre-OST0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 7272.946902] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7272.950737] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7272.962474] LustreError: Skipped 1 previous similar message [ 7272.995710] Lustre: Skipped 4 previous similar messages [ 7275.387359] Lustre: server umount lustre-OST0001 complete [ 7276.159832] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7276.185129] LustreError: Skipped 2 previous similar messages [ 7292.385779] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7292.386038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7292.394530] LustreError: Skipped 1 previous similar message [ 7292.396987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7292.404553] Lustre: Skipped 1 previous similar message [ 7292.425791] Lustre: Skipped 3 previous similar messages [ 7293.416332] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7293.429823] LustreError: Skipped 7 previous similar messages [ 7298.414437] Lustre: server umount lustre-OST0000 complete [ 7303.648160] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7303.651050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7303.667526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7303.683086] Lustre: Skipped 1 previous similar message [ 7303.699451] Lustre: Skipped 3 previous similar messages [ 7311.783049] LustreError: 259748:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196270 with bad export cookie 4906328747019438605 [ 7311.789335] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7311.792505] LustreError: 259748:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7326.175149] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7326.498355] Lustre: server umount lustre-MDT0001 complete [ 7326.514276] Lustre: Skipped 1 previous similar message [ 7337.745206] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7340.463728] Key type lgssc unregistered [ 7340.697286] LNet: 264957:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7340.710903] LNet: Removed LNI 192.168.203.115@tcp [ 7341.742448] Key type .llcrypt unregistered [ 7341.744724] Key type ._llcrypt unregistered [ 7376.879775] alg: No test for adler32 (adler32-zlib) [ 7377.642239] Key type ._llcrypt registered [ 7377.646353] Key type .llcrypt registered [ 7377.788194] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 7393.434903] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7395.104952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7395.692299] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 7395.964091] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7395.970550] LNet: Accept secure, port 988 [ 7397.681107] Key type lgssc registered [ 7398.834691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7410.726402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7417.679452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7423.731456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7430.185961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7436.231559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7436.290521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7437.600874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7437.633438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7437.732733] Lustre: lustre-MDT0000: new disk, initializing [ 7437.827890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7437.851137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7442.266701] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7452.334091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7452.429553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.518939] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7452.565299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7452.567630] Lustre: Skipped 1 previous similar message [ 7452.644553] Lustre: lustre-MDT0001: new disk, initializing [ 7452.743331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7452.790950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7452.805901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7457.239239] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7467.333486] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7472.462274] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7478.302664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7478.380975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7478.395389] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7478.673380] Lustre: lustre-OST0000: new disk, initializing [ 7478.677384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7478.748860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7483.355552] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7486.522785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7486.538856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7493.446543] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7498.727934] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7499.024913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7503.574315] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7503.850744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7507.428368] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7507.434231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7507.438038] LustreError: Skipped 1 previous similar message [ 7507.455113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7507.465816] Lustre: Skipped 1 previous similar message [ 7512.568419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7512.580580] Lustre: Skipped 1 previous similar message [ 7517.685467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7517.695016] Lustre: Skipped 1 previous similar message [ 7520.224083] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7520.437825] Lustre: server umount lustre-OST0000 complete [ 7524.320899] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7524.329582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7524.342526] Lustre: Skipped 1 previous similar message [ 7524.348061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7527.904828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7530.405563] Lustre: server umount lustre-MDT0000 complete [ 7533.026931] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7533.048254] LustreError: Skipped 1 previous similar message [ 7533.928749] LustreError: 269069:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196492 with bad export cookie 12733510576612216840 [ 7533.944414] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7533.952332] LustreError: 269069:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7534.313843] Lustre: server umount lustre-MDT0001 complete [ 7542.879930] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 15:55:01 (1776196501) [ 7548.968159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7549.509683] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7549.649928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7555.043351] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7556.156809] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7560.164275] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7565.282254] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7565.672365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7566.053253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7571.109272] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7578.749194] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7584.138765] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7590.250596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7590.635644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7595.308989] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7602.181862] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7616.481027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.484311] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7616.521227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7616.528073] Lustre: Skipped 3 previous similar messages [ 7621.565260] Lustre: server umount lustre-OST0000 complete [ 7625.588370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7637.284448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7637.702492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7641.754898] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7648.997429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7649.027020] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7649.041919] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.115@tcp (at 0@lo) [ 7649.052216] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 7649.057698] Lustre: Skipped 1 previous similar message [ 7649.068179] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 7649.839104] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7664.608529] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7664.622547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7664.630057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7664.656713] Lustre: Skipped 2 previous similar messages [ 7667.613254] Lustre: server umount lustre-OST0000 complete [ 7673.827540] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7673.837073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7677.662957] Lustre: server umount lustre-MDT0000 complete [ 7679.968259] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7679.982931] LustreError: Skipped 1 previous similar message [ 7681.820908] LustreError: 273612:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196640 with bad export cookie 12733510576612217694 [ 7681.827818] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7681.829659] LustreError: 273612:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7694.653108] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7697.824920] Key type lgssc unregistered [ 7698.155438] LNet: 279050:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7698.174620] LNet: Removed LNI 192.168.203.115@tcp [ 7699.028272] Key type .llcrypt unregistered [ 7699.036881] Key type ._llcrypt unregistered [ 7721.037078] alg: No test for adler32 (adler32-zlib) [ 7721.791749] Key type ._llcrypt registered [ 7721.795409] Key type .llcrypt registered [ 7721.911118] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7722.832594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7723.056791] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 7723.272190] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7723.280696] LNet: Accept secure, port 988 [ 7724.951400] Key type lgssc registered [ 7725.926544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7733.016385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7734.512980] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7734.572392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.381459] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7739.874849] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7744.993164] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7746.645124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7747.042042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7751.603244] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7759.784614] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7765.516329] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7771.878459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7772.264788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7776.752317] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7784.353986] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7784.555851] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 7805.418763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7805.419126] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7805.421384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7805.442019] Lustre: Skipped 1 previous similar message [ 7810.529780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7810.540161] Lustre: Skipped 2 previous similar messages [ 7810.827438] Lustre: server umount lustre-OST0000 complete [ 7815.652822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.653872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7815.669832] Lustre: Skipped 1 previous similar message [ 7815.685369] Lustre: Skipped 1 previous similar message [ 7820.198114] Lustre: server umount lustre-MDT0000 complete [ 7820.768375] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.776534] LustreError: Skipped 1 previous similar message [ 7823.173777] LustreError: 280498:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196782 with bad export cookie 17577577079729315200 [ 7823.184411] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7823.500719] Lustre: server umount lustre-MDT0001 complete [ 7834.265619] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7836.600342] Key type lgssc unregistered [ 7836.804400] LNet: 284942:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7836.811856] LNet: Removed LNI 192.168.203.115@tcp [ 7837.398053] Key type .llcrypt unregistered [ 7837.405991] Key type ._llcrypt unregistered [ 7849.937135] alg: No test for adler32 (adler32-zlib) [ 7850.690547] Key type ._llcrypt registered [ 7850.692611] Key type .llcrypt registered [ 7850.797922] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7851.754386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7852.305781] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 7852.487844] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7852.491562] LNet: Accept secure, port 988 [ 7854.127215] Key type lgssc registered [ 7855.282636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7862.266260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7863.678235] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7863.755405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7867.973546] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7868.899408] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7874.023853] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7875.362575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7875.669984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7879.858455] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7887.446897] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7892.974951] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7898.597802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7898.892940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7903.292430] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 7911.497704] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7913.268685] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 7930.848397] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7930.865547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7930.911345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7931.786040] Lustre: server umount lustre-OST0000 complete [ 7936.992404] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7937.003871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7937.024770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7939.041763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7939.051556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7942.111697] Lustre: 289241:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196895/real 1776196895] req@000000003a3f274c x1862477347307456/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776196901 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7944.162157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7944.184484] Lustre: Skipped 2 previous similar messages [ 7944.387970] Lustre: server umount lustre-MDT0000 complete [ 7945.185169] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7945.197321] LustreError: Skipped 1 previous similar message [ 7948.985392] LustreError: 286190:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196907 with bad export cookie 16316510410383559852 [ 7948.999078] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7949.003997] LustreError: 286190:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7949.250388] Lustre: server umount lustre-MDT0001 complete [ 7962.134769] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7964.563276] Key type lgssc unregistered [ 7964.845514] LNet: 290048:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7964.857790] LNet: Removed LNI 192.168.203.115@tcp [ 7965.563749] Key type .llcrypt unregistered [ 7965.566386] Key type ._llcrypt unregistered [ 7982.118112] alg: No test for adler32 (adler32-zlib) [ 7982.878377] Key type ._llcrypt registered [ 7982.889225] Key type .llcrypt registered [ 7983.018407] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7984.195083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7984.672392] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 7984.856032] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7984.864308] LNet: Accept secure, port 988 [ 7986.511119] Key type lgssc registered [ 7987.790784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7995.444217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7996.865795] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7996.933522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8001.851979] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8002.018151] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8007.147247] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8010.191769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.624338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8014.678191] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8021.481767] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8026.712799] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8032.734554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8033.086059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8037.660589] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8044.766256] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8048.433814] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 8058.853245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8058.857105] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8058.867596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8058.883316] Lustre: Skipped 1 previous similar message [ 8063.972183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8063.979236] Lustre: Skipped 1 previous similar message [ 8064.108885] Lustre: server umount lustre-OST0000 complete [ 8069.088800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8069.115499] Lustre: Skipped 1 previous similar message [ 8069.120904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8069.131863] Lustre: Skipped 1 previous similar message [ 8074.215242] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8074.235592] LustreError: Skipped 1 previous similar message [ 8074.265824] Lustre: server umount lustre-MDT0000 complete [ 8078.675881] LustreError: 291314:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197037 with bad export cookie 5298620632490451053 [ 8078.689289] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8078.695901] LustreError: 291314:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8079.002821] Lustre: server umount lustre-MDT0001 complete [ 8092.246839] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8095.061995] Key type lgssc unregistered [ 8095.372611] LNet: 295012:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8095.379589] LNet: Removed LNI 192.168.203.115@tcp [ 8096.088480] Key type .llcrypt unregistered [ 8096.094061] Key type ._llcrypt unregistered [ 8119.457417] alg: No test for adler32 (adler32-zlib) [ 8120.219515] Key type ._llcrypt registered [ 8120.223865] Key type .llcrypt registered [ 8120.342728] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8121.240310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8121.974468] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 8122.221094] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8122.226590] LNet: Accept secure, port 988 [ 8123.951161] Key type lgssc registered [ 8124.865405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8130.858795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8132.325701] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.387686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8136.360359] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8137.697119] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8142.816983] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8144.294600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8144.673468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8149.160504] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8157.447543] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8163.338825] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8169.417810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8169.661084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8174.587387] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8182.969137] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 8184.943804] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8210.915832] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8210.926372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8210.948431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8213.985405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8213.991200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8213.992630] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8214.017589] Lustre: Skipped 1 previous similar message [ 8215.840937] Lustre: server umount lustre-OST0000 complete [ 8221.664649] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8221.676063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8221.704393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8224.224330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8224.227927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8225.928415] Lustre: server umount lustre-MDT0000 complete [ 8229.350971] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8229.366408] LustreError: Skipped 1 previous similar message [ 8230.010768] LustreError: 296454:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197189 with bad export cookie 11365328958977430768 [ 8230.018257] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8230.025194] LustreError: 296454:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8230.455584] Lustre: server umount lustre-MDT0001 complete [ 8242.859807] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8245.443125] Key type lgssc unregistered [ 8245.678546] LNet: 300954:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8245.698186] LNet: Removed LNI 192.168.203.115@tcp [ 8246.553743] Key type .llcrypt unregistered [ 8246.555913] Key type ._llcrypt unregistered [ 8262.145604] alg: No test for adler32 (adler32-zlib) [ 8262.904639] Key type ._llcrypt registered [ 8262.909492] Key type .llcrypt registered [ 8263.046937] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8263.956420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8264.360915] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 8264.532677] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8264.537684] LNet: Accept secure, port 988 [ 8266.201516] Key type lgssc registered [ 8267.022836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8274.301133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8275.851656] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8275.920674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8280.736527] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8281.057767] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8286.176501] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8289.409293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.822821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8295.352492] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8304.347456] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8312.531640] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8319.801000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8320.171694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8324.536499] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8326.313258] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 8332.496497] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8355.300163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8355.320525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8355.329355] Lustre: Skipped 1 previous similar message [ 8359.119461] Lustre: server umount lustre-OST0000 complete [ 8365.540984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8365.550850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8365.561589] Lustre: Skipped 2 previous similar messages [ 8365.578594] Lustre: Skipped 1 previous similar message [ 8370.660972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8370.675645] Lustre: Skipped 1 previous similar message [ 8376.806101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8376.813401] Lustre: Skipped 3 previous similar messages [ 8377.311267] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8377.449898] Lustre: server umount lustre-MDT0000 complete [ 8380.986145] LustreError: 302201:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197339 with bad export cookie 12330776506863642661 [ 8380.996742] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8381.007446] LustreError: 302201:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8381.355916] Lustre: server umount lustre-MDT0001 complete [ 8392.941522] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8395.741982] Key type lgssc unregistered [ 8396.010572] LNet: 306087:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8396.021760] LNet: Removed LNI 192.168.203.115@tcp [ 8396.744101] Key type .llcrypt unregistered [ 8396.751191] Key type ._llcrypt unregistered [ 8413.030786] alg: No test for adler32 (adler32-zlib) [ 8413.786911] Key type ._llcrypt registered [ 8413.797085] Key type .llcrypt registered [ 8413.943511] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8415.229335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8415.710583] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 8415.931077] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8415.945471] LNet: Accept secure, port 988 [ 8417.591184] Key type lgssc registered [ 8418.618754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8426.644098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8428.172821] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8428.215709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8433.638536] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8434.756832] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8438.753651] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8443.873183] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8445.753841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8446.155968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8451.946679] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8461.957894] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8470.998470] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8479.174227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8479.665977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.715880] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8493.710733] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8503.215456] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 8513.512234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8513.518848] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8513.521117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8513.532101] Lustre: Skipped 1 previous similar message [ 8518.738127] Lustre: server umount lustre-OST0000 complete [ 8523.749389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8523.752923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8523.783642] Lustre: Skipped 1 previous similar message [ 8523.794645] Lustre: Skipped 1 previous similar message [ 8528.864320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8528.868341] Lustre: Skipped 1 previous similar message [ 8529.087328] Lustre: server umount lustre-MDT0000 complete [ 8529.902142] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8529.918623] LustreError: Skipped 1 previous similar message [ 8532.990217] LustreError: 307353:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197491 with bad export cookie 11180737924783294362 [ 8532.997719] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8533.005359] LustreError: 307353:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8533.283859] Lustre: server umount lustre-MDT0001 complete [ 8546.383729] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8549.060776] Key type lgssc unregistered [ 8549.375443] LNet: 311054:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8549.386632] LNet: Removed LNI 192.168.203.115@tcp [ 8550.210978] Key type .llcrypt unregistered [ 8550.216487] Key type ._llcrypt unregistered [ 8563.125356] alg: No test for adler32 (adler32-zlib) [ 8563.886558] Key type ._llcrypt registered [ 8563.889396] Key type .llcrypt registered [ 8564.032897] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8582.534552] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 8595.356686] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8596.271674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8597.104771] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 8597.356786] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8597.361757] LNet: Accept secure, port 988 [ 8599.095123] Key type lgssc registered [ 8600.177791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8608.943257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8615.920763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8622.563090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8629.421517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8637.861937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8637.942472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8639.334630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8639.405886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8639.529905] Lustre: lustre-MDT0000: new disk, initializing [ 8639.629536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8639.651639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8644.596130] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8654.987352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8655.045200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8655.111272] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8655.145441] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8655.149464] Lustre: Skipped 1 previous similar message [ 8655.201479] Lustre: lustre-MDT0001: new disk, initializing [ 8655.268724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8655.297406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8655.309301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8660.314682] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8670.477873] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8676.503966] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8682.281817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8682.358221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8682.364623] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8682.666602] Lustre: lustre-OST0000: new disk, initializing [ 8682.678415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8682.790913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8687.141970] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8692.320703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8692.328503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8697.100489] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8702.995494] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8703.331578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8709.541911] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8709.796147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8713.193220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8713.208660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8713.210195] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8713.221032] Lustre: Skipped 1 previous similar message [ 8713.244136] Lustre: Skipped 1 previous similar message [ 8718.503299] Lustre: server umount lustre-OST0000 complete [ 8723.937745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8723.943607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8723.958493] Lustre: Skipped 1 previous similar message [ 8723.966018] Lustre: Skipped 1 previous similar message [ 8728.815283] Lustre: server umount lustre-MDT0000 complete [ 8729.066502] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8730.091530] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8730.109647] LustreError: Skipped 2 previous similar messages [ 8733.412534] LustreError: 314802:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197692 with bad export cookie 6312251499703495448 [ 8733.418047] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8733.423613] LustreError: 314802:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8733.602025] Lustre: server umount lustre-MDT0001 complete [ 8744.250368] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:15:02 (1776197702) [ 8751.041771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.617766] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8751.670288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8756.738285] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8757.241972] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8761.829698] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8767.471510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8767.907500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8772.648419] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8780.954596] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8786.745155] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8793.473268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8793.919615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8799.595705] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8807.751844] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8818.656737] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8818.668781] LustreError: Skipped 1 previous similar message [ 8818.674385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8818.702619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8821.729941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8821.733259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8821.762462] Lustre: Skipped 1 previous similar message [ 8823.837851] Lustre: server umount lustre-OST0000 complete [ 8829.408325] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8829.413184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8829.434950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8842.209032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8842.214317] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8842.220589] Lustre: Skipped 5 previous similar messages [ 8842.344556] Lustre: server umount lustre-MDT0000 complete [ 8846.694530] LustreError: 319343:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197805 with bad export cookie 6312251499703496302 [ 8846.701279] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8846.712207] LustreError: 319343:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8858.039981] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8861.061603] Key type lgssc unregistered [ 8861.316652] LNet: 323022:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8861.325409] LNet: Removed LNI 192.168.203.115@tcp [ 8862.433523] Key type .llcrypt unregistered [ 8862.437911] Key type ._llcrypt unregistered [ 8889.471153] alg: No test for adler32 (adler32-zlib) [ 8890.229929] Key type ._llcrypt registered [ 8890.235066] Key type .llcrypt registered [ 8890.347356] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 8902.712559] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8903.806963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8904.250497] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 8904.446534] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8904.455484] LNet: Accept secure, port 988 [ 8906.119211] Key type lgssc registered [ 8907.243224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8916.338380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8923.250243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8929.287895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8935.835666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8940.806197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8946.299240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8952.016284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8952.069753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8953.299966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8953.344206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8953.456277] Lustre: lustre-MDT0000: new disk, initializing [ 8953.565772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8953.589302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8957.830390] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8966.351679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8966.423610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8966.496549] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8966.524921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8966.532102] Lustre: Skipped 1 previous similar message [ 8966.608033] Lustre: lustre-MDT0001: new disk, initializing [ 8966.714496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8966.749409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8966.761559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8970.624643] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 8979.016424] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8984.187095] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8990.366335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8990.485654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8990.494176] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8990.766357] Lustre: lustre-OST2710: new disk, initializing [ 8990.771497] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 8990.839428] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8995.511238] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9000.546095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9000.562199] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9005.524268] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9013.199971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9013.302771] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9013.315728] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 9013.451567] Lustre: lustre-OST03e8: new disk, initializing [ 9013.457888] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 9013.577949] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9018.487285] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9019.948741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9019.956528] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9026.498860] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9033.180344] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 9033.408324] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9037.956288] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 9038.177276] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9042.871933] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 9043.102530] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9047.923690] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 9048.191900] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9053.664238] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9053.670772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9053.690541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9055.256316] Lustre: server umount lustre-MDT0000 complete [ 9055.726081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9055.730768] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9055.783370] LustreError: Skipped 3 previous similar messages [ 9059.330790] LustreError: 326967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198018 with bad export cookie 2797132482589036915 [ 9059.337084] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9059.347033] LustreError: 326967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9059.706241] Lustre: server umount lustre-MDT0001 complete [ 9069.535795] Lustre: 332465:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198022/real 1776198022] req@000000004f940ce2 x1862478450425920/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776198028 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9069.588623] Lustre: server umount lustre-OST2710 complete [ 9079.263177] Lustre: 332667:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198032/real 1776198032] req@00000000e3acfcab x1862478450426368/t0(0) o39->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776198038 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9079.538917] Lustre: server umount lustre-OST03e8 complete [ 9099.405770] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 9107.688824] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9117.598785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9124.960808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9132.307078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9139.288271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9146.424317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9146.473712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9146.663581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9146.695778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9146.757174] Lustre: lustre-MDT0000: new disk, initializing [ 9146.827618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9146.844908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9151.750212] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9161.670804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9161.751224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9161.864327] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9161.930562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9161.936037] Lustre: Skipped 1 previous similar message [ 9162.047249] Lustre: lustre-MDT0001: new disk, initializing [ 9162.161271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9162.210962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9162.227540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9167.270431] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9177.519995] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9183.799784] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9190.277580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9190.385164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9190.390952] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9190.810951] Lustre: lustre-OST0000: new disk, initializing [ 9190.826065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9190.933121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9195.565716] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9196.632456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9196.638529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9205.145614] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9210.398761] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9210.855494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9217.269647] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9217.660272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9222.624874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9222.632647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9222.633890] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9222.656877] Lustre: Skipped 3 previous similar messages [ 9222.694789] Lustre: Skipped 1 previous similar message [ 9226.354119] Lustre: server umount lustre-OST0000 complete [ 9232.874755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9232.881412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9232.889773] Lustre: Skipped 1 previous similar message [ 9232.910837] Lustre: Skipped 1 previous similar message [ 9236.678810] Lustre: server umount lustre-MDT0000 complete [ 9237.984931] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9238.002994] LustreError: Skipped 1 previous similar message [ 9239.012134] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9239.031701] LustreError: Skipped 1 previous similar message [ 9241.069062] LustreError: 335575:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198200 with bad export cookie 2797132482589038532 [ 9241.073745] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9241.085778] LustreError: 335575:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9251.130629] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:23:29 (1776198209) [ 9252.985414] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9254.876674] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:23:32 (1776198212) [ 9258.670550] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9266.305337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9270.776696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9275.088851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9279.005709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9280.022450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9286.750706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9286.968541] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9286.990796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9287.194876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9291.430311] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9299.656298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.779944] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9299.806485] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9304.607324] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9311.540652] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9316.803879] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9323.036931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9323.108298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9323.313062] Lustre: Denying initial registration attempt from nid 192.168.203.115@tcp, specified as failover [ 9323.320192] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9323.330490] LustreError: 343603:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 9323.335835] LustreError: 343603:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 9323.341098] LustreError: 343603:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 9323.439384] Lustre: server umount lustre-OST0000 complete [ 9323.441936] Lustre: Skipped 1 previous similar message [ 9323.444646] LustreError: 343603:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9330.658938] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9330.669672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9330.691902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9337.709295] LustreError: 341644:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198296 with bad export cookie 2797132482589039386 [ 9337.714545] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9337.721294] LustreError: 341644:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9350.498415] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9354.386766] Key type lgssc unregistered [ 9354.882181] LNet: 344679:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9354.924096] LNet: Removed LNI 192.168.203.115@tcp [ 9356.078777] Key type .llcrypt unregistered [ 9356.082466] Key type ._llcrypt unregistered [ 9373.196309] alg: No test for adler32 (adler32-zlib) [ 9373.970651] Key type ._llcrypt registered [ 9373.990402] Key type .llcrypt registered [ 9374.247466] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9376.181204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9376.805080] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 9377.095498] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9377.100948] LNet: Accept secure, port 988 [ 9378.855225] Key type lgssc registered [ 9380.833762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9389.987868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9394.673677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9398.521292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9402.235251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9403.132668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9416.782272] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9425.438619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9426.721345] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9426.757400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9427.009163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9431.588550] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9438.905609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.980600] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9439.002954] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9439.185223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9443.692215] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9451.497330] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9457.437722] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9463.519702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9463.613273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9463.815822] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9463.998282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9468.207101] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9475.490983] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9481.185796] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9481.199492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9481.214713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9483.668367] Lustre: server umount lustre-OST0000 complete [ 9486.816531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9486.819708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9486.827842] Lustre: Skipped 2 previous similar messages [ 9486.845959] Lustre: Skipped 2 previous similar messages [ 9491.939238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9491.955290] Lustre: Skipped 1 previous similar message [ 9492.863115] Lustre: server umount lustre-MDT0000 complete [ 9496.484580] LustreError: 347554:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198455 with bad export cookie 15648776803724346216 [ 9496.489170] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9496.498883] LustreError: 347554:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9496.759841] Lustre: server umount lustre-MDT0001 complete [ 9506.164245] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9508.434252] Key type lgssc unregistered [ 9508.725895] LNet: 351256:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9508.738374] LNet: Removed LNI 192.168.203.115@tcp [ 9509.394498] Key type .llcrypt unregistered [ 9509.396195] Key type ._llcrypt unregistered [ 9529.971322] alg: No test for adler32 (adler32-zlib) [ 9530.729484] Key type ._llcrypt registered [ 9530.731528] Key type .llcrypt registered [ 9530.849616] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9531.636766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9531.958711] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 9532.115387] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9532.131070] LNet: Accept secure, port 988 [ 9533.767369] Key type lgssc registered [ 9534.541187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9542.992646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.409123] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9544.509100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9549.205277] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9549.803726] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9554.913831] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9558.990272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9559.013196] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9559.630458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9565.221759] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9575.747185] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9582.320668] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9589.765930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9590.230696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9594.587613] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9601.904596] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9619.951980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9619.985977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9619.988473] Lustre: Skipped 2 previous similar messages [ 9620.119697] Lustre: server umount lustre-MDT0000 complete [ 9623.483052] LustreError: 352700:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198582 with bad export cookie 11083980039043808215 [ 9623.487974] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9623.494449] LustreError: 352700:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9624.202954] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9624.233382] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 9625.057343] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9625.072173] Lustre: Skipped 2 previous similar messages [ 9628.064665] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9628.070822] Lustre: Skipped 1 previous similar message [ 9629.837216] Lustre: server umount lustre-MDT0001 complete [ 9635.855726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9643.252650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9656.803908] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0x99d2351ebcb9fbd7 to 0x99d2351ebcba1816 [ 9656.816579] Lustre: MGC192.168.203.115@tcp: Connection restored to 192.168.203.115@tcp (at 0@lo) [ 9657.137454] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9657.161311] LustreError: Skipped 4 previous similar messages [ 9657.274823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9657.350457] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 9659.304708] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 9661.959567] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9671.370083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9671.701764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9676.921700] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9680.942092] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 9685.347351] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9686.783334] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9692.414086] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9693.919885] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9698.275638] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9698.286742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9698.302906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9698.307728] Lustre: Skipped 2 previous similar messages [ 9701.350868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9704.401721] Lustre: server umount lustre-OST0000 complete [ 9711.587660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9711.590862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9711.609870] Lustre: Skipped 1 previous similar message [ 9711.625852] Lustre: Skipped 3 previous similar messages [ 9714.769959] Lustre: server umount lustre-MDT0000 complete [ 9716.708336] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9716.735826] LustreError: Skipped 6 previous similar messages [ 9718.274454] LustreError: 352702:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198677 with bad export cookie 11083980039043815446 [ 9718.277424] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9718.285059] LustreError: 352702:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9730.086253] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9732.545159] Key type lgssc unregistered [ 9732.834106] LNet: 359245:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9732.852785] LNet: Removed LNI 192.168.203.115@tcp [ 9733.614930] Key type .llcrypt unregistered [ 9733.620179] Key type ._llcrypt unregistered [ 9755.647805] alg: No test for adler32 (adler32-zlib) [ 9756.401658] Key type ._llcrypt registered [ 9756.407695] Key type .llcrypt registered [ 9756.568949] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9757.896686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9758.270943] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [ 9758.450518] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9758.458474] LNet: Accept secure, port 988 [ 9760.087178] Key type lgssc registered [ 9761.392655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9771.276692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9772.681415] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9772.746892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9777.466406] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9778.145069] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9783.265045] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9786.073372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9786.431869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9790.778257] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9798.084428] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9803.705843] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9806.816468] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9806.826251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9806.846568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9808.868416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9808.871549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9812.782853] Lustre: server umount lustre-MDT0000 complete [ 9813.988524] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9814.009406] LustreError: Skipped 1 previous similar message [ 9816.017842] LustreError: 360692:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198775 with bad export cookie 17016798504401751347 [ 9816.026442] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9816.036693] LustreError: 360692:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9816.331543] Lustre: server umount lustre-MDT0001 complete [ 9830.291503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9830.569875] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9830.606515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9830.768476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9834.829945] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9843.535400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.680744] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9843.720391] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9843.944833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9848.151135] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9854.818556] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9860.308073] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9866.959938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9867.137666] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9867.155757] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9867.164364] LustreError: 365403:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 9867.176784] LustreError: 365403:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 9867.187670] LustreError: 365403:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 9867.329690] Lustre: server umount lustre-OST0000 complete [ 9867.333445] LustreError: 365403:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9873.005870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9873.179081] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9873.394219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9877.392974] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9881.222787] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 9884.124897] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9890.375094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9890.421671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9890.426737] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 9890.470341] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9890.523928] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 9890.534388] Lustre: Skipped 1 previous similar message [ 9890.541604] Lustre: lustre-OST0001: new disk, initializing [ 9890.546570] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 9890.629786] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9894.108946] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [ 9895.932848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9895.945494] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9900.329227] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9906.149641] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9906.156515] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9906.169092] LustreError: Skipped 1 previous similar message [ 9906.171267] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9906.200477] Lustre: Skipped 1 previous similar message [ 9911.277583] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9911.284144] Lustre: Skipped 2 previous similar messages [ 9917.407130] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9917.449311] Lustre: server umount lustre-OST0001 complete [ 9921.506773] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9921.511326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9921.516750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9921.516759] Lustre: Skipped 2 previous similar messages [ 9921.532842] LustreError: Skipped 1 previous similar message [ 9921.548167] Lustre: Skipped 1 previous similar message [ 9931.750825] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9931.761916] LustreError: Skipped 3 previous similar messages [ 9935.327240] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9935.484609] Lustre: server umount lustre-OST0000 complete [ 9941.472262] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9941.479943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9941.490710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9941.495688] Lustre: Skipped 5 previous similar messages [ 9944.726873] Lustre: server umount lustre-MDT0000 complete [ 9948.381057] LustreError: 365787:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198907 with bad export cookie 17016798504401751900 [ 9948.388159] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9948.400752] LustreError: 365787:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9959.425491] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9962.059781] Key type lgssc unregistered [ 9962.397820] LNet: 368783:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9962.414666] LNet: Removed LNI 192.168.203.115@tcp [ 9963.141637] Key type .llcrypt unregistered [ 9963.144475] Key type ._llcrypt unregistered [10003.469654] alg: No test for adler32 (adler32-zlib) [10004.228380] Key type ._llcrypt registered [10004.233926] Key type .llcrypt registered [10004.431315] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [10018.865931] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10019.659595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10020.293879] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [10020.557867] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [10020.563727] LNet: Accept secure, port 988 [10022.199171] Key type lgssc registered [10023.198622] Lustre: Echo OBD driver; http://www.lustre.org/ [10030.753314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10036.425561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10041.771065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10047.422822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10054.703822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10054.751638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10055.914996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10055.940690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10056.027604] Lustre: lustre-MDT0000: new disk, initializing [10056.113696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10056.129661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10062.167866] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10073.943308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10074.007875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10074.092945] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10074.153826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10074.162137] Lustre: Skipped 1 previous similar message [10074.264822] Lustre: lustre-MDT0001: new disk, initializing [10074.375712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10074.430235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10074.445693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10078.802756] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10088.680715] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10094.546951] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10101.807179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10101.883682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10101.890799] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10102.146748] Lustre: lustre-OST0000: new disk, initializing [10102.159720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10102.240101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10107.667920] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10112.090219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10112.112838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10118.344548] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10123.045820] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10123.318142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10129.469114] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10129.907757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10132.964608] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10132.971542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10132.977021] LustreError: Skipped 1 previous similar message [10132.981989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10133.007343] Lustre: Skipped 1 previous similar message [10133.048996] Lustre: Skipped 1 previous similar message [10138.096735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10138.111028] Lustre: Skipped 1 previous similar message [10138.566395] Lustre: server umount lustre-OST0000 complete [10143.201682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10143.213659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10143.222376] Lustre: Skipped 1 previous similar message [10143.245541] Lustre: Skipped 1 previous similar message [10148.258905] Lustre: server umount lustre-MDT0000 complete [10148.328032] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10152.007144] LustreError: 372896:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199111 with bad export cookie 9039911181255815329 [10152.009032] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10152.019762] LustreError: 372896:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10152.340932] Lustre: server umount lustre-MDT0001 complete [10160.522946] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 16:38:38 (1776199118) [10165.930919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10171.482848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10192.749165] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [10201.703759] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10211.510881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10217.642127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10222.983535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10228.491694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10234.780644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10234.837418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10235.028994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10235.052916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10235.130303] Lustre: lustre-MDT0000: new disk, initializing [10235.208693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10235.226817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10239.209705] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10249.067734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10249.144696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10249.225608] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10249.273536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10249.279155] Lustre: Skipped 1 previous similar message [10249.376702] Lustre: lustre-MDT0001: new disk, initializing [10249.467354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10249.508791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10249.526648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10253.616484] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10262.487524] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10268.877922] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10276.944477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10276.998906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10277.004734] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10277.175383] Lustre: lustre-OST0000: new disk, initializing [10277.178434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10277.300487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10282.485185] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10283.091041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10283.102890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10291.678212] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10296.448555] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10296.744449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10301.082376] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10301.301832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10303.972164] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10303.974414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10303.980203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10303.987789] Lustre: Skipped 1 previous similar message [10303.993113] Lustre: Skipped 1 previous similar message [10309.093466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10309.109545] Lustre: Skipped 1 previous similar message [10317.279696] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10317.377721] Lustre: server umount lustre-OST0000 complete [10321.377182] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10321.384096] LustreError: Skipped 1 previous similar message [10321.387307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10321.396700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10321.401484] Lustre: Skipped 2 previous similar messages [10326.389040] Lustre: server umount lustre-MDT0000 complete [10329.257108] LustreError: 381520:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199288 with bad export cookie 9039911181255816190 [10329.257897] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10329.269801] LustreError: 381520:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10336.113734] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 16:41:34 (1776199294) [10344.250755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10344.634631] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10344.649411] LustreError: Skipped 1 previous similar message [10344.719800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10348.797828] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10350.050625] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10355.175630] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10356.269972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10360.597914] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10366.839956] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10372.237433] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10378.151485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10378.450090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10378.455782] Lustre: Skipped 1 previous similar message [10382.914279] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10391.208399] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10404.470835] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [10406.210894] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [10407.920605] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [10409.734235] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [10411.931559] Lustre: Failing over lustre-MDT0000 [10412.145577] Lustre: server umount lustre-MDT0000 complete [10412.148074] Lustre: Skipped 1 previous similar message [10412.735175] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.025414] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10413.042698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10413.073338] Lustre: Skipped 1 previous similar message [10421.729393] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10421.766158] LustreError: Skipped 8 previous similar messages [10423.711180] Lustre: 371121:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199375/real 1776199375] req@00000000063978d2 x1862479620658688/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776199382 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10423.766903] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10433.074684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10440.161683] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10440.172263] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0x7d743752e7d5b754 to 0x7d743752e7d5bb44 [10440.174254] LustreError: Skipped 19 previous similar messages [10440.210609] Lustre: MGC192.168.203.115@tcp: Connection restored to 192.168.203.115@tcp (at 0@lo) [10440.503740] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10442.365000] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10445.454445] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10445.808859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [10445.828804] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10445.861327] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [10454.471167] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10455.948609] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10459.050690] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [10460.884781] Lustre: Failing over lustre-MDT0000 [10461.042804] Lustre: server umount lustre-MDT0000 complete [10461.153144] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10461.154600] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10461.185320] Lustre: Skipped 4 previous similar messages [10473.107093] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10473.134072] LustreError: Skipped 11 previous similar messages [10480.262336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10480.396815] LustreError: 11-0: MGC192.168.203.115@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [10480.410839] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10480.432893] Lustre: Evicted from MGS (at 192.168.203.115@tcp) after server handle changed from 0x7d743752e7d5bb44 to 0x7d743752e7d5be46 [10480.448679] Lustre: MGC192.168.203.115@tcp: Connection restored to (at 0@lo) [10480.451774] Lustre: Skipped 2 previous similar messages [10480.681939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10480.692798] Lustre: Skipped 1 previous similar message [10480.745716] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10483.330106] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10485.039188] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10485.737416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [10485.767497] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10485.828243] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [10494.763212] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10496.536165] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10506.211398] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10506.216328] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10506.221869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10506.248298] Lustre: Skipped 2 previous similar messages [10506.254846] Lustre: Skipped 3 previous similar messages [10508.652974] Lustre: server umount lustre-MDT0000 complete [10512.393989] LustreError: 385303:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199471 with bad export cookie 9039911181255818822 [10512.397160] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10512.409637] LustreError: 385303:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10522.079146] Lustre: 391136:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199475/real 1776199475] req@000000006ddbd22e x1862479620679808/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776199481 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10535.887225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10540.504087] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10541.538379] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10541.558995] LustreError: Skipped 11 previous similar messages [10548.741614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.515111] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10560.468369] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10565.916639] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10572.391116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10577.247121] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10583.943944] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10586.166766] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [10594.275567] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10594.284491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10594.300612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10594.303751] Lustre: Skipped 1 previous similar message [10608.095341] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10608.279640] Lustre: server umount lustre-OST0000 complete [10608.283969] Lustre: Skipped 2 previous similar messages [10623.116350] LustreError: 391943:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199582 with bad export cookie 9039911181255819557 [10623.117671] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10623.130492] LustreError: 391943:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10635.245699] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [10637.837122] Key type lgssc unregistered [10638.107120] LNet: 395618:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10638.116336] LNet: Removed LNI 192.168.203.115@tcp [10638.935591] Key type .llcrypt unregistered [10638.943713] Key type ._llcrypt unregistered [10662.388671] alg: No test for adler32 (adler32-zlib) [10663.147191] Key type ._llcrypt registered [10663.151742] Key type .llcrypt registered [10663.292937] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10664.665559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10665.061528] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [10665.267037] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [10665.270319] LNet: Accept secure, port 988 [10666.896729] Key type lgssc registered [10667.715542] Lustre: Echo OBD driver; http://www.lustre.org/ [10674.774849] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10674.784832] LustreError: 397105:0:(osd_handler.c:8215:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10674.795782] LustreError: 397105:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [10674.800891] LustreError: 397105:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10674.811414] LustreError: 397105:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10674.817109] LustreError: 397105:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [10682.187406] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10682.194460] LustreError: 397503:0:(osd_handler.c:8215:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10682.203684] LustreError: 397503:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [10682.209186] LustreError: 397503:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10682.214347] LustreError: 397503:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10682.220894] LustreError: 397503:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [10700.593541] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [10704.393610] Key type lgssc unregistered [10704.786818] LNet: 398461:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10704.799240] LNet: Removed LNI 192.168.203.115@tcp [10705.962948] Key type .llcrypt unregistered [10705.968278] Key type ._llcrypt unregistered [10727.215334] alg: No test for adler32 (adler32-zlib) [10727.983501] Key type ._llcrypt registered [10727.985316] Key type .llcrypt registered [10728.060077] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [10742.186060] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10742.901268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10743.587817] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [10743.769746] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [10743.776262] LNet: Accept secure, port 988 [10745.415145] Key type lgssc registered [10746.350924] Lustre: Echo OBD driver; http://www.lustre.org/ [10754.652925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10761.441374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10767.821032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10775.164097] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10782.236467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10782.361406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10783.590695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10783.628271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10783.705819] Lustre: lustre-MDT0000: new disk, initializing [10783.813595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10783.852421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10788.755205] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10800.772409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10800.849344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.974075] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10801.039469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10801.054175] Lustre: Skipped 1 previous similar message [10801.223332] Lustre: lustre-MDT0001: new disk, initializing [10801.334538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10801.409133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10801.426691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10806.182648] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10816.189168] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10822.476312] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10828.657232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10828.732412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10828.738367] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10828.969859] Lustre: lustre-OST0000: new disk, initializing [10828.975630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10829.064169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10833.362705] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10836.552092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10836.564114] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10843.346737] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10848.402796] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10848.621294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10852.920851] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10853.124889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10857.444960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10857.451363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10857.458858] Lustre: Skipped 1 previous similar message [10857.467946] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10862.562662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10862.571083] Lustre: Skipped 2 previous similar messages [10867.685076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10867.694973] Lustre: Skipped 1 previous similar message [10869.215271] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10869.495824] Lustre: server umount lustre-OST0000 complete [10877.925346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10877.956852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10877.964947] Lustre: Skipped 1 previous similar message [10879.640518] Lustre: server umount lustre-MDT0000 complete [10883.040530] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.062299] LustreError: Skipped 1 previous similar message [10883.668611] LustreError: 402616:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199842 with bad export cookie 11368846931335619511 [10883.677226] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10883.682344] LustreError: 402616:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10883.972805] Lustre: server umount lustre-MDT0001 complete [10893.247952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10895.209668] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 16:50:53 (1776199853) [10901.611327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10901.995855] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10902.049602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10906.217909] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10907.110152] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10907.126368] LustreError: Skipped 1 previous similar message [10912.228740] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10914.487885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10914.796174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10918.550153] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10925.464154] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10931.077645] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10938.149939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10938.636455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10944.718319] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10954.783599] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10962.418859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10962.531287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10962.543624] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [10962.806483] Lustre: lustre-OST0001: new disk, initializing [10962.810899] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [10962.909721] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10967.422651] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [10970.245345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10970.250887] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10976.305762] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10980.849792] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10980.851060] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10980.852741] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10980.857241] LustreError: Skipped 1 previous similar message [10980.892033] Lustre: Skipped 2 previous similar messages [10985.711844] Lustre: server umount lustre-OST0001 complete [10985.956109] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10985.999184] LustreError: Skipped 1 previous similar message [10996.200169] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10996.224214] LustreError: Skipped 4 previous similar messages [11004.389115] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11004.400843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11004.424596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11004.427826] Lustre: Skipped 2 previous similar messages [11006.436799] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11008.676262] Lustre: server umount lustre-OST0000 complete [11016.675571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11016.688770] Lustre: Skipped 1 previous similar message [11018.433416] Lustre: server umount lustre-MDT0000 complete [11021.793369] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11021.818215] LustreError: Skipped 5 previous similar messages [11022.157142] LustreError: 406347:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199981 with bad export cookie 11368846931335620365 [11022.162105] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11022.171771] LustreError: 406347:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11039.573458] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [11042.369346] Key type lgssc unregistered [11042.611584] LNet: 411278:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11042.625269] LNet: Removed LNI 192.168.203.115@tcp [11043.460289] Key type .llcrypt unregistered [11043.465988] Key type ._llcrypt unregistered [11075.893031] alg: No test for adler32 (adler32-zlib) [11076.648545] Key type ._llcrypt registered [11076.653194] Key type .llcrypt registered [11076.743742] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [11090.982851] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [11092.473248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11092.996272] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [11093.316506] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [11093.321316] LNet: Accept secure, port 988 [11095.015313] Key type lgssc registered [11095.945797] Lustre: Echo OBD driver; http://www.lustre.org/ [11105.480094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11112.227230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11119.163363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11125.803901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11132.806914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11132.899681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11134.180533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11134.211760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11134.329734] Lustre: lustre-MDT0000: new disk, initializing [11134.471885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11134.531701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11140.250522] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11153.518880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11153.794263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11154.083731] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11154.172145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11154.191293] Lustre: Skipped 1 previous similar message [11154.329465] Lustre: lustre-MDT0001: new disk, initializing [11154.610352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11154.724202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11154.752794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11161.128860] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11173.865964] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11182.157642] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11190.957937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11191.107229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11191.127256] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11191.549259] Lustre: lustre-OST0000: new disk, initializing [11191.558856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11191.718138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11197.077487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11197.106729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11197.866123] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11208.540957] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11213.179751] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [11213.543536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11218.131454] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [11218.385284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11223.008298] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11223.009110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11223.024725] LustreError: Skipped 1 previous similar message [11223.025895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11223.055499] Lustre: Skipped 1 previous similar message [11226.695472] Lustre: server umount lustre-OST0000 complete [11231.203861] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11231.213636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11231.233849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11231.236823] Lustre: Skipped 1 previous similar message [11233.251654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11233.252972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11236.299831] Lustre: server umount lustre-MDT0000 complete [11238.368152] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11238.380364] LustreError: Skipped 1 previous similar message [11239.400788] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11239.423158] LustreError: Skipped 1 previous similar message [11240.424052] LustreError: 415393:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200199 with bad export cookie 3471945213005222830 [11240.434374] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11240.443770] LustreError: 415393:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [11240.914713] Lustre: server umount lustre-MDT0001 complete [11251.004584] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 16:56:48 (1776200208) [11261.634457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11270.544343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11271.011067] Lustre: lustre-MDT0000: new disk, initializing [11271.060610] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11271.168267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11275.995391] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11276.260363] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11281.376954] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11283.715732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11284.081171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11288.524718] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11296.683881] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11303.048137] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11306.466233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11306.467897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11306.483681] Lustre: Skipped 1 previous similar message [11306.492165] Lustre: Skipped 1 previous similar message [11311.257088] Lustre: server umount lustre-MDT0000 complete [11311.586937] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11311.601853] LustreError: Skipped 1 previous similar message [11315.572123] LustreError: 420530:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200274 with bad export cookie 3471945213005223684 [11315.581142] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11315.585938] LustreError: 420530:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11315.930533] Lustre: server umount lustre-MDT0001 complete [11325.365455] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 16:58:03 (1776200283) [11331.031905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11331.532445] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11331.561422] LustreError: Skipped 2 previous similar messages [11331.629275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11336.210574] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11344.477892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11348.888332] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11356.321544] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11362.313545] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11368.837401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11369.299049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11369.307308] Lustre: Skipped 1 previous similar message [11374.760901] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11384.531616] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11391.730250] Lustre: Permanently deactivating lustre-OST0000 [11391.743825] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [11401.784931] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11403.366234] LustreError: 426244:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [11403.376256] LustreError: 426244:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [11403.384443] LustreError: 426244:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [11406.861586] LustreError: 426293:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [11406.867108] LustreError: 426293:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [11415.311251] Lustre: server umount lustre-OST0000 complete [11418.375390] LustreError: 426543:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [11418.383170] LustreError: 426543:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [11420.640777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11420.643747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11420.661583] Lustre: Skipped 1 previous similar message [11420.665477] Lustre: Skipped 1 previous similar message [11426.783415] Lustre: 426640:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200379/real 1776200379] req@00000000de8dc2e5 x1862480745776128/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776200385 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11430.880625] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11430.896556] LustreError: Skipped 3 previous similar messages [11432.320146] LustreError: 426641:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200391 with bad export cookie 3471945213005224237 [11432.333745] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11432.336965] LustreError: 426641:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [11438.559238] Lustre: 426843:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200391/real 1776200391] req@0000000088f7c5f1 x1862480745777280/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776200397 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11439.311033] Lustre: server umount lustre-MDT0001 complete [11439.324185] Lustre: Skipped 1 previous similar message [11454.935145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11459.905594] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11463.119056] LustreError: 427753:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [11469.721878] Lustre: 428100:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.115@tcp [11477.983211] Lustre: 428199:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200430/real 1776200430] req@000000005344e49d x1862480745777856/t0(0) o251->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776200436 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11478.075631] Lustre: server umount MGS complete [11489.979482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11490.529940] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11490.549522] Lustre: Skipped 1 previous similar message [11490.623354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11495.117289] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11495.904241] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11495.922770] LustreError: Skipped 1 previous similar message [11504.794511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11505.011037] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11510.915947] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11519.512600] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11525.727843] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11527.877693] Lustre: Permanently reactivating lustre-OST0000 [11527.899185] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [11527.908523] Lustre: Skipped 2 previous similar messages [11557.855128] LustreError: 430761:0:(osp_dev.c:684:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11564.839241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11565.071619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11565.077824] Lustre: Skipped 1 previous similar message [11569.103631] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11576.935915] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11587.554405] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11587.557226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11587.568498] LustreError: Skipped 1 previous similar message [11587.569749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11587.569755] Lustre: Skipped 3 previous similar messages [11587.614358] Lustre: Skipped 1 previous similar message [11588.512102] Lustre: server umount lustre-OST0000 complete [11592.161375] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11602.213816] LustreError: 428809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200561 with bad export cookie 3471945213005225189 [11602.218588] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11602.229938] LustreError: 428809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11615.889834] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [11618.800155] Key type lgssc unregistered [11619.160794] LNet: 433321:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11619.172896] LNet: Removed LNI 192.168.203.115@tcp [11620.087422] Key type .llcrypt unregistered [11620.098278] Key type ._llcrypt unregistered [11638.595846] alg: No test for adler32 (adler32-zlib) [11639.348608] Key type ._llcrypt registered [11639.352782] Key type .llcrypt registered [11639.461762] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [11653.589368] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [11654.625792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11655.108297] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [11655.394569] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [11655.398674] LNet: Accept secure, port 988 [11657.055146] Key type lgssc registered [11658.108160] Lustre: Echo OBD driver; http://www.lustre.org/ [11666.728052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11672.693170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11678.583617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11684.664368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11691.767205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11691.875347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11693.262545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11693.301250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11693.432395] Lustre: lustre-MDT0000: new disk, initializing [11693.587370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11693.610310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11699.074524] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11710.785869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11710.879882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11710.934635] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11710.965862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11710.969831] Lustre: Skipped 1 previous similar message [11711.037247] Lustre: lustre-MDT0001: new disk, initializing [11711.118696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11711.152221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11711.166432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11716.127558] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11725.984386] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11731.790194] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11737.626887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11737.688393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11737.700206] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11737.955681] Lustre: lustre-OST0000: new disk, initializing [11737.961777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11738.052136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11742.922816] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11746.874772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11746.890446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11752.211839] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11757.145458] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [11757.376235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11761.661554] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [11761.871767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11767.777953] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11767.789887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11767.800468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11767.825500] Lustre: Skipped 1 previous similar message [11769.971662] Lustre: server umount lustre-OST0000 complete [11778.017109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11778.032136] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11778.036243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11778.036254] Lustre: Skipped 1 previous similar message [11778.049938] Lustre: Skipped 1 previous similar message [11780.432872] Lustre: server umount lustre-MDT0000 complete [11783.137306] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11783.144238] LustreError: Skipped 1 previous similar message [11784.163916] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11784.175744] LustreError: Skipped 1 previous similar message [11784.413777] LustreError: 436623:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200743 with bad export cookie 9936887552848063064 [11784.423673] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11784.430069] LustreError: 436623:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [11784.774745] Lustre: server umount lustre-MDT0001 complete [11794.898952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11797.726953] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:05:55 (1776200755) [11806.174127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11806.833182] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11806.934274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11812.331517] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11812.552050] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11817.445584] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.031568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11822.345177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11826.962475] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11834.273850] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11839.811588] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11846.379133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11846.794760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11852.047939] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11860.496465] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11863.304081] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11866.455544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11878.889627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11878.918754] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11878.921146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11878.949489] Lustre: Skipped 2 previous similar messages [11881.070316] Lustre: server umount lustre-OST0000 complete [11889.120503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11889.128985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11889.144160] Lustre: Skipped 2 previous similar messages [11889.145793] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11889.145802] LustreError: Skipped 1 previous similar message [11889.185594] Lustre: Skipped 1 previous similar message [11894.240367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11894.256571] Lustre: Skipped 1 previous similar message [11900.383512] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [11900.555419] Lustre: server umount lustre-MDT0000 complete [11904.999101] LustreError: 441211:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200863 with bad export cookie 9936887552848063918 [11905.013222] LustreError: 441211:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [11905.021543] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11918.685857] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [11921.640858] Key type lgssc unregistered [11921.924067] LNet: 445079:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11921.939669] LNet: Removed LNI 192.168.203.115@tcp [11922.750625] Key type .llcrypt unregistered [11922.763963] Key type ._llcrypt unregistered [11947.607843] alg: No test for adler32 (adler32-zlib) [11948.368675] Key type ._llcrypt registered [11948.371960] Key type .llcrypt registered [11948.523681] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [11948.895350] Key type .llcrypt unregistered [11948.900254] Key type ._llcrypt unregistered [11965.149477] alg: No test for adler32 (adler32-zlib) [11965.904402] Key type ._llcrypt registered [11965.908426] Key type .llcrypt registered [11966.019694] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [11967.185373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11967.553237] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [11967.792803] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [11967.799543] LNet: Accept secure, port 988 [11969.473088] Key type lgssc registered [11970.552728] Lustre: Echo OBD driver; http://www.lustre.org/ [11978.659378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11980.359118] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.520113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11985.887757] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11986.278833] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [11991.010475] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11995.117924] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11995.918712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11996.415736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12001.687473] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12001.762234] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12001.790468] LustreError: Skipped 3 previous similar messages [12010.018148] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12010.041777] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [12010.046018] LustreError: Skipped 3 previous similar messages [12011.771226] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12020.558279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12020.998905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12027.100615] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12040.672544] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12040.684693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12040.712503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12041.708386] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12041.714279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12041.716301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12041.759311] Lustre: Skipped 1 previous similar message [12043.372640] Lustre: server umount lustre-OST0000 complete [12051.936805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12051.949527] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12051.951175] Lustre: Skipped 1 previous similar message [12051.956672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12051.989451] Lustre: Skipped 1 previous similar message [12053.707194] Lustre: server umount lustre-MDT0000 complete [12057.060171] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12057.070491] LustreError: Skipped 4 previous similar messages [12057.555770] LustreError: 447393:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201016 with bad export cookie 12785771393880601231 [12057.562338] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12057.580727] LustreError: 447393:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [12057.975932] Lustre: server umount lustre-MDT0001 complete [12069.097380] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12071.376894] Key type lgssc unregistered [12071.620727] LNet: 450679:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12071.634964] LNet: Removed LNI 192.168.203.115@tcp [12072.337711] Key type .llcrypt unregistered [12072.340510] Key type ._llcrypt unregistered [12091.907535] alg: No test for adler32 (adler32-zlib) [12092.681389] Key type ._llcrypt registered [12092.683194] Key type .llcrypt registered [12092.768670] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12093.694842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12093.996845] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [12094.193988] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12094.202286] LNet: Accept secure, port 988 [12095.895364] Key type lgssc registered [12096.795879] Lustre: Echo OBD driver; http://www.lustre.org/ [12102.898324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12120.479748] LustreError: 452114:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 1: rc = -5 [12151.199208] LustreError: 452114:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [12181.983633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12186.729156] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12194.075494] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12201.514607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12203.810929] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12203.907836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12203.979375] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [12208.539247] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12209.123278] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12214.245100] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12216.038793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12216.378913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12220.406106] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12228.081334] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12233.845632] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12240.886436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12240.888318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12240.903262] Lustre: Skipped 1 previous similar message [12240.935427] Lustre: Skipped 1 previous similar message [12245.986722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12245.997989] Lustre: Skipped 1 previous similar message [12251.109191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12251.119439] Lustre: Skipped 1 previous similar message [12255.201091] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [12255.315802] Lustre: server umount lustre-OST0000 complete [12261.345279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12261.347962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12261.359939] Lustre: Skipped 1 previous similar message [12261.369413] Lustre: Skipped 1 previous similar message [12265.265215] Lustre: server umount lustre-MDT0000 complete [12266.464643] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12266.479556] LustreError: Skipped 1 previous similar message [12268.935152] LustreError: 452121:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201227 with bad export cookie 6132116154992848433 [12268.951590] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12269.305805] Lustre: server umount lustre-MDT0001 complete [12281.041485] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12283.640776] Key type lgssc unregistered [12283.871280] LNet: 455809:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12283.878128] LNet: Removed LNI 192.168.203.115@tcp [12284.551635] Key type .llcrypt unregistered [12284.553725] Key type ._llcrypt unregistered [12305.560915] alg: No test for adler32 (adler32-zlib) [12306.318183] Key type ._llcrypt registered [12306.322118] Key type .llcrypt registered [12306.426339] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12307.357136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12307.855540] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [12308.039531] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12308.047141] LNet: Accept secure, port 988 [12309.679259] Key type lgssc registered [12310.707392] Lustre: Echo OBD driver; http://www.lustre.org/ [12317.332543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12318.707264] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12318.747750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12322.698831] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12323.812168] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12328.938316] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12329.644335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12330.037762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12333.778256] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12340.342781] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12346.233851] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12353.539698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12354.241232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12359.448849] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12368.569353] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [12368.851168] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12373.988264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12373.996148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12374.016830] Lustre: Skipped 2 previous similar messages [12374.017253] Lustre: Skipped 1 previous similar message [12375.703466] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) [12375.712594] Lustre: Skipped 1 previous similar message [12378.737245] Lustre: server umount lustre-MDT0000 complete [12379.108433] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12379.121987] LustreError: Skipped 2 previous similar messages [12381.023907] LustreError: 457255:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.15@tcp arrived at 1776201340 with bad export cookie 6676975131485660902 [12381.045359] LustreError: 457255:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [12382.543837] LustreError: 457255:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201341 with bad export cookie 6676975131485659999 [12382.549784] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12382.555810] LustreError: 457255:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [12384.233182] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12384.233453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12384.241966] LustreError: Skipped 1 previous similar message [12384.266439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12387.297229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12388.829290] Lustre: server umount lustre-OST0000 complete [12393.441639] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12393.450093] LustreError: Skipped 5 previous similar messages [12395.425976] Lustre: server umount lustre-MDT0001 complete [12406.060903] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12408.393845] Key type lgssc unregistered [12408.584090] LNet: 461081:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12408.595493] LNet: Removed LNI 192.168.203.115@tcp [12409.289077] Key type .llcrypt unregistered [12409.293741] Key type ._llcrypt unregistered [12431.055173] alg: No test for adler32 (adler32-zlib) [12431.808774] Key type ._llcrypt registered [12431.811906] Key type .llcrypt registered [12431.920554] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12432.881465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12433.161371] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [12433.365982] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12433.369616] LNet: Accept secure, port 988 [12435.047163] Key type lgssc registered [12436.144081] Lustre: Echo OBD driver; http://www.lustre.org/ [12443.051472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12444.457713] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12444.497331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12448.621280] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12449.767486] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12454.881875] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12457.113058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12457.494824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12461.585530] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12468.780442] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12474.177273] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12480.246357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12480.682884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12484.876111] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12492.859649] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12495.028937] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:129 [12503.721917] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [12505.568269] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12505.574877] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12505.588220] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12505.597030] Lustre: Skipped 1 previous similar message [12505.603317] Lustre: Skipped 1 previous similar message [12507.309086] Lustre: server umount lustre-MDT0001 complete [12508.802838] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12515.809739] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12515.810065] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12515.810320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12515.812156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12515.824351] LustreError: Skipped 2 previous similar messages [12520.185328] Lustre: server umount lustre-OST0000 complete [12531.167189] Lustre: 465596:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201484/real 1776201484] req@000000007a65cad9 x1862482150833664/t0(0) o251->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776201490 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12531.411931] Lustre: server umount lustre-MDT0000 complete [12544.979866] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12547.890564] Key type lgssc unregistered [12548.189419] LNet: 466352:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12548.205469] LNet: Removed LNI 192.168.203.115@tcp [12548.951949] Key type .llcrypt unregistered [12548.955767] Key type ._llcrypt unregistered [12578.701521] alg: No test for adler32 (adler32-zlib) [12579.456380] Key type ._llcrypt registered [12579.458134] Key type .llcrypt registered [12579.620534] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [12594.906656] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12595.543214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12596.178085] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [12596.341899] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12596.347148] LNet: Accept secure, port 988 [12597.999193] Key type lgssc registered [12599.258773] Lustre: Echo OBD driver; http://www.lustre.org/ [12608.079857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12614.397860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12620.436173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12626.206276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12634.778420] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12643.227360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12649.728521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12649.792190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12651.092843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [12651.121454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [12651.215110] Lustre: lustre-MDT0000: new disk, initializing [12651.330125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12651.365507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12656.162967] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12664.541298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12670.730712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12670.793709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12670.799327] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [12671.054705] Lustre: lustre-OST0000: new disk, initializing [12671.061703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [12671.076433] Lustre: Skipped 1 previous similar message [12671.127524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12675.325547] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12684.409877] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12692.515183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12698.780194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12698.878803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12698.976519] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12699.038521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [12699.214271] Lustre: lustre-MDT0001: new disk, initializing [12699.282738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12699.290244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12699.370224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12703.542554] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12711.924190] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [12712.153681] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12715.967667] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [12716.177078] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12722.855455] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [12723.680858] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12723.688303] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12723.692206] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12723.698671] Lustre: Skipped 1 previous similar message [12726.043296] Lustre: server umount lustre-MDT0001 complete [12727.934581] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12728.808693] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12728.830363] LustreError: Skipped 1 previous similar message [12733.921449] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12733.943887] LustreError: Skipped 1 previous similar message [12739.043348] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12739.065297] LustreError: Skipped 1 previous similar message [12744.161432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12744.163519] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12744.186759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12744.210341] Lustre: Skipped 1 previous similar message [12749.284205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12754.400367] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12754.401364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12754.419447] LustreError: Skipped 1 previous similar message [12757.983239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [12758.117411] Lustre: server umount lustre-OST0000 complete [12773.855164] Lustre: 474786:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201726/real 1776201726] req@00000000f8f59098 x1862482321754752/t0(0) o251->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776201732 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12774.053617] Lustre: server umount lustre-MDT0000 complete [12788.804988] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12791.491406] Key type lgssc unregistered [12791.832871] LNet: 475545:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12791.848509] LNet: Removed LNI 192.168.203.115@tcp [12792.899797] Key type .llcrypt unregistered [12792.909803] Key type ._llcrypt unregistered [12834.691793] alg: No test for adler32 (adler32-zlib) [12835.450976] Key type ._llcrypt registered [12835.455540] Key type .llcrypt registered [12835.598454] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12835.875923] Key type .llcrypt unregistered [12835.881455] Key type ._llcrypt unregistered [12853.164323] alg: No test for adler32 (adler32-zlib) [12853.922157] Key type ._llcrypt registered [12853.928072] Key type .llcrypt registered [12854.038667] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12855.246118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12855.692689] Lustre: Lustre: Build Version: 2.15.8_11_g8969407 [12855.898297] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12855.902626] LNet: Accept secure, port 988 [12857.547317] Key type lgssc registered [12858.701716] Lustre: Echo OBD driver; http://www.lustre.org/ [12867.145343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12872.882336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12878.298685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12884.810885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12884.861612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12886.044658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [12886.072925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [12886.141832] Lustre: lustre-MDT0000: new disk, initializing [12886.204359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12886.231819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12890.079559] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12900.043729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12900.113398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12900.216252] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12900.260149] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [12900.265223] Lustre: Skipped 1 previous similar message [12900.338272] Lustre: lustre-MDT0001: new disk, initializing [12900.455744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12900.500295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12900.521990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12904.505903] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12914.505308] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12920.779639] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12927.521163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12927.610224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12927.621482] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [12927.898939] Lustre: lustre-OST0000: new disk, initializing [12927.902922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [12927.982880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12932.071624] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [12937.783221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12937.799831] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12941.109859] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12948.453222] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12948.462079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12948.472682] Lustre: Skipped 1 previous similar message [12948.485285] Lustre: Skipped 2 previous similar messages [12953.568984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12953.576147] Lustre: Skipped 2 previous similar messages [12958.697023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12958.707090] Lustre: Skipped 2 previous similar messages [12961.247453] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [12961.390768] Lustre: server umount lustre-MDT0000 complete [12963.811437] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12963.843595] LustreError: Skipped 2 previous similar messages [12965.481737] LustreError: 479061:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201924 with bad export cookie 11842730170099308254 [12965.485730] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12965.496922] LustreError: 479061:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [12965.872300] Lustre: server umount lustre-MDT0001 complete [12975.071320] Lustre: 482324:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201928/real 1776201928] req@00000000009a4e17 x1862482593334592/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201934 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12975.199925] Lustre: server umount lustre-OST0000 complete [12986.662806] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 17:25:44 (1776201944) [12987.625939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12994.212072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12994.702932] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12994.820551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12999.678949] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [13000.163126] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13005.284308] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13009.096044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13009.471749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13013.899525] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [13022.839225] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13029.240098] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13036.362963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13036.462815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13036.803497] Lustre: Found index 0 for lustre-OST0000, updating log [13037.090352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13041.986291] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [13045.298320] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [13048.582698] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13054.439229] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13054.451303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13054.460713] Lustre: Skipped 1 previous similar message [13054.465803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13059.373046] Lustre: server umount lustre-OST0000 complete [13065.696398] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13065.708554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13065.716641] Lustre: Skipped 1 previous similar message [13065.720965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13065.728556] Lustre: Skipped 1 previous similar message [13069.523786] Lustre: server umount lustre-MDT0000 complete [13072.352615] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13072.374220] LustreError: Skipped 1 previous similar message [13073.307153] LustreError: 483130:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202032 with bad export cookie 11842730170099309591 [13073.318278] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13073.324409] LustreError: 483130:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [13086.690735] LustreError: 486216:0:(import.c:355:ptlrpc_invalidate_import()) lustre-OST0000_UUID: timeout waiting for callback (1 != 0) [13086.709475] LustreError: 486216:0:(import.c:378:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000669332a4 x1862482593352512/t0(0) o8->lustre-OST0000-osc-MDT0001@1.2.3.4@tcp:28/4 lens 520/544 e 0 to 0 dl 1776202042 ref 2 fl UnregRPC:ENU/0/ffffffff rc -5/-1 job:'' [13086.736559] LustreError: 486216:0:(import.c:389:ptlrpc_invalidate_import()) lustre-OST0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting for them to error out. [13089.692731] Lustre: server umount lustre-MDT0001 complete [13098.004790] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 17:27:36 (1776202056) [13104.085701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13110.208415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13117.102373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13124.244735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13143.075624] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [13154.434654] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [13165.313592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13173.760687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13181.895635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13190.058944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13197.841360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13197.920269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13198.264489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [13198.313292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [13198.413726] Lustre: lustre-MDT0000: new disk, initializing [13198.559155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13198.596074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13203.814670] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [13215.360718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13215.417804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13215.487823] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [13215.531057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [13215.540994] Lustre: Skipped 1 previous similar message [13215.655488] Lustre: lustre-MDT0001: new disk, initializing [13215.857112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13215.872177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13219.917612] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [13228.906917] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13234.350774] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13241.229861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13241.304784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13241.312378] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [13241.577831] Lustre: lustre-OST0000: new disk, initializing [13241.586763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [13241.692794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13241.697961] Lustre: Skipped 1 previous similar message [13246.255049] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all 8 [13251.712934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13251.725661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13258.019118] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13263.873870] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13264.150752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13269.675339] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [13270.001450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13272.545863] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13272.555536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13272.571795] LustreError: Skipped 1 previous similar message [13272.575506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13272.575514] Lustre: Skipped 2 previous similar messages [13272.646191] Lustre: Skipped 2 previous similar messages [13286.367133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [13286.557212] Lustre: server umount lustre-OST0000 complete [13292.512055] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13292.522214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13292.535046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13292.541144] Lustre: Skipped 5 previous similar messages [13298.144495] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13298.164603] LustreError: Skipped 1 previous similar message [13299.821131] LustreError: 490177:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202258 with bad export cookie 11842730170099310655 [13299.828536] LustreError: 490177:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [13299.830755] LustreError: 166-1: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13322.688776] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [13330.291544] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [13339.860639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13347.833295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13354.237704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13360.828415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13362.407196] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13113 sec ========= 17:32:00 (1776202320) [13408.073367] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13410.487083] Key type lgssc unregistered [13410.770173] LNet: 498922:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13410.787472] LNet: Removed LNI 192.168.203.115@tcp [13411.541590] Key type .llcrypt unregistered [13411.545535] Key type ._llcrypt unregistered