[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512281175 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003262] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008536] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009038] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011119] LSM: Security Framework initializing [ 0.012037] Yama: becoming mindful. [ 0.013026] SELinux: Initializing. [ 0.014050] *** VALIDATE selinux *** [ 0.021759] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026068] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028061] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030093] *** VALIDATE tmpfs *** [ 0.032264] *** VALIDATE proc *** [ 0.033212] *** VALIDATE cgroup *** [ 0.034006] *** VALIDATE cgroup2 *** [ 0.036159] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039023] Spectre V2 : User space: Vulnerable [ 0.040004] Speculative Store Bypass: Vulnerable [ 0.043655] debug: unmapping init [mem 0xffffffffb6e59000-0xffffffffb6e60fff] [ 0.045843] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046775] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047021] ... version: 2 [ 0.048007] ... bit width: 48 [ 0.049008] ... generic registers: 4 [ 0.050006] ... value mask: 0000ffffffffffff [ 0.051010] ... max period: 00007fffffffffff [ 0.052009] ... fixed-purpose events: 3 [ 0.053006] ... event mask: 000000070000000f [ 0.055224] rcu: Hierarchical SRCU implementation. [ 0.057759] smp: Bringing up secondary CPUs ... [ 0.058585] x86: Booting SMP configuration: [ 0.059032] .... node #0, CPUs: #1 #2 #3 [ 0.068624] smp: Brought up 1 node, 4 CPUs [ 0.070011] smpboot: Max logical packages: 1 [ 0.071018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.190579] node 0 deferred pages initialised in 117ms [ 0.194169] devtmpfs: initialized [ 0.195295] x86/mm: Memory block size: 128MB [ 0.197599] gcov: version magic: 0x41383552 [ 0.199248] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.200077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.201235] pinctrl core: initialized pinctrl subsystem [ 0.202214] [ 0.202747] ************************************************************* [ 0.203010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.204010] ** ** [ 0.205009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.206008] ** ** [ 0.207005] ** This means that this kernel is built to expose internal ** [ 0.208009] ** IOMMU data structures, which may compromise security on ** [ 0.209009] ** your system. ** [ 0.210009] ** ** [ 0.211009] ** If you see this message and you are not debugging the ** [ 0.212010] ** kernel, report this immediately to your vendor! ** [ 0.213007] ** ** [ 0.214012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.215010] ************************************************************* [ 0.216758] NET: Registered protocol family 16 [ 0.217456] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.218047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.219046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.220439] cpuidle: using governor menu [ 0.223150] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.226321] PCI: Using configuration type 1 for base access [ 0.229160] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.238079] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.239057] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.241031] cryptd: max_cpu_qlen set to 1000 [ 0.242239] ACPI: Added _OSI(Module Device) [ 0.243011] ACPI: Added _OSI(Processor Device) [ 0.244009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.245007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248437] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251391] ACPI: Interpreter enabled [ 0.252067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.253013] ACPI: Using IOAPIC for interrupt routing [ 0.255130] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258389] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266654] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269023] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.272069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.276049] acpiphp: Slot [2] registered [ 0.278137] acpiphp: Slot [3] registered [ 0.279073] acpiphp: Slot [4] registered [ 0.281072] acpiphp: Slot [5] registered [ 0.282126] acpiphp: Slot [6] registered [ 0.283100] acpiphp: Slot [7] registered [ 0.285125] acpiphp: Slot [8] registered [ 0.286090] acpiphp: Slot [9] registered [ 0.287078] acpiphp: Slot [10] registered [ 0.288085] acpiphp: Slot [11] registered [ 0.290076] acpiphp: Slot [12] registered [ 0.291072] acpiphp: Slot [13] registered [ 0.292077] acpiphp: Slot [14] registered [ 0.293077] acpiphp: Slot [15] registered [ 0.294067] acpiphp: Slot [16] registered [ 0.296067] acpiphp: Slot [17] registered [ 0.297062] acpiphp: Slot [18] registered [ 0.298064] acpiphp: Slot [19] registered [ 0.301124] acpiphp: Slot [20] registered [ 0.303061] acpiphp: Slot [21] registered [ 0.305153] acpiphp: Slot [22] registered [ 0.307072] acpiphp: Slot [23] registered [ 0.308074] acpiphp: Slot [24] registered [ 0.309094] acpiphp: Slot [25] registered [ 0.310067] acpiphp: Slot [26] registered [ 0.311109] acpiphp: Slot [27] registered [ 0.313075] acpiphp: Slot [28] registered [ 0.314074] acpiphp: Slot [29] registered [ 0.316097] acpiphp: Slot [30] registered [ 0.317125] acpiphp: Slot [31] registered [ 0.318065] PCI host bridge to bus 0000:00 [ 0.319017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.323019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.326035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.327019] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.330014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331270] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.335475] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.340615] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.351527] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.354013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362549] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.366067] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.369072] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.371707] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.376015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.387023] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.391021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.395310] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.404021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.413031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.431017] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.438689] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.445017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.456015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.477016] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.487110] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.495017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.505018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.518015] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.527919] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.533016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.540068] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.557024] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.573495] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.580026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.588023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.602016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.620102] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.637017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.644015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.661019] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.671000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.678385] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.680304] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.685230] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.687199] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.695043] iommu: Default domain type: Passthrough [ 0.699570] SCSI subsystem initialized [ 0.702186] ACPI: bus type USB registered [ 0.703132] usbcore: registered new interface driver usbfs [ 0.706088] usbcore: registered new interface driver hub [ 0.708083] usbcore: registered new device driver usb [ 0.711158] pps_core: LinuxPPS API ver. 1 registered [ 0.714021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.720058] PTP clock support registered [ 0.725125] EDAC MC: Ver: 3.0.0 [ 0.727127] PCI: Using ACPI for IRQ routing [ 0.728558] NetLabel: Initializing [ 0.729011] NetLabel: domain hash size = 128 [ 0.735023] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.737092] NetLabel: unlabeled traffic allowed by default [ 0.739109] vgaarb: loaded [ 0.740292] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.741014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.748591] clocksource: Switched to clocksource kvm-clock [ 0.905035] VFS: Disk quotas dquot_6.6.0 [ 0.908894] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.912178] *** VALIDATE ramfs *** [ 0.914632] *** VALIDATE hugetlbfs *** [ 0.916256] pnp: PnP ACPI init [ 0.920269] pnp: PnP ACPI: found 6 devices [ 0.942849] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.946501] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.948565] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.950512] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.952727] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.954960] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.957436] NET: Registered protocol family 2 [ 0.959675] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.964382] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.967442] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.972346] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.975469] TCP: Hash tables configured (established 65536 bind 65536) [ 0.978224] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.980841] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.983287] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.985854] NET: Registered protocol family 1 [ 0.989167] RPC: Registered named UNIX socket transport module. [ 0.991231] RPC: Registered udp transport module. [ 0.992665] RPC: Registered tcp transport module. [ 0.994169] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.996265] NET: Registered protocol family 44 [ 0.997700] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.999696] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.001611] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.003756] PCI: CLS 0 bytes, default 64 [ 1.005245] Unpacking initramfs... [ 2.554359] debug: unmapping init [mem 0xffff98e67cc54000-0xffff98e67ffbffff] [ 2.559260] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.561134] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.564473] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.094063] Initialise system trusted keyrings [ 3.095414] Key type blacklist registered [ 3.096925] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.105537] zbud: loaded [ 3.108208] *** VALIDATE nfs *** [ 3.109156] *** VALIDATE nfs4 *** [ 3.110392] pstore: using deflate compression [ 3.115014] Platform Keyring initialized [ 3.258464] NET: Registered protocol family 38 [ 3.260399] Key type asymmetric registered [ 3.262704] Asymmetric key parser 'x509' registered [ 3.264416] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.269049] io scheduler mq-deadline registered [ 3.270538] io scheduler kyber registered [ 3.273238] io scheduler bfq registered [ 3.275851] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.279171] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.285048] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.287730] ACPI: Power Button [PWRF] [ 3.445872] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.557115] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.813266] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.930241] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.179685] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.216475] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.256106] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.266599] Non-volatile memory driver v1.3 [ 4.267995] Linux agpgart interface v0.103 [ 4.302661] virtio_blk virtio1: [vda] 67960 512-byte logical blocks (34.8 MB/33.2 MiB) [ 4.310599] vda: detected capacity change from 0 to 34795520 [ 4.333402] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.336867] vdb: detected capacity change from 0 to 1073741824 [ 4.354407] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.360783] vdc: detected capacity change from 0 to 2621440000 [ 4.376105] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.379516] vdd: detected capacity change from 0 to 2621440000 [ 4.398307] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.401183] vde: detected capacity change from 0 to 4294967296 [ 4.420475] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.423473] vdf: detected capacity change from 0 to 4294967296 [ 4.431981] libphy: Fixed MDIO Bus: probed [ 4.439324] usbcore: registered new interface driver usbserial_generic [ 4.441831] usbserial: USB Serial support registered for generic [ 4.444133] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.453325] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.455073] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.461654] mousedev: PS/2 mouse device common for all mice [ 4.466146] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.468669] rtc_cmos 00:05: RTC can wake from S4 [ 4.474696] rtc_cmos 00:05: registered as rtc0 [ 4.476544] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.480155] intel_pstate: CPU model not supported [ 4.481144] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.492979] hid: raw HID events driver (C) Jiri Kosina [ 4.495120] usbcore: registered new interface driver usbhid [ 4.497349] usbhid: USB HID core driver [ 4.498855] drop_monitor: Initializing network drop monitor service [ 4.500954] Initializing XFRM netlink socket [ 4.503165] NET: Registered protocol family 10 [ 4.503906] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.509105] Segment Routing with IPv6 [ 4.513208] NET: Registered protocol family 17 [ 4.515117] mpls_gso: MPLS GSO support [ 4.531158] RAS: Correctable Errors collector initialized. [ 4.533319] AVX version of gcm_enc/dec engaged. [ 4.538681] AES CTR mode by8 optimization enabled [ 4.683684] sched_clock: Marking stable (4683665429, 0)->(5612424829, -928759400) [ 4.686505] registered taskstats version 1 [ 4.688927] Loading compiled-in X.509 certificates [ 4.690495] zswap: loaded using pool lzo/zbud [ 4.728088] Key type big_key registered [ 4.744506] Key type encrypted registered [ 4.746631] ima: No TPM chip found, activating TPM-bypass! [ 4.749400] ima: Allocated hash algorithm: sha1 [ 4.750814] ima: No architecture policies found [ 4.752752] evm: Initialising EVM extended attributes: [ 4.754704] evm: security.selinux [ 4.755704] evm: security.ima [ 4.756610] evm: security.capability [ 4.757686] evm: HMAC attrs: 0x1 [ 4.760912] rtc_cmos 00:05: setting system clock to 2025-10-07 13:10:26 UTC (1759842626) [ 4.767173] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 4.770326] debug: unmapping init [mem 0xffffffffb6b82000-0xffffffffb6e58fff] [ 4.781096] Write protecting the kernel read-only data: 28672k [ 4.798078] debug: unmapping init [mem 0xffffffffb5203000-0xffffffffb53fffff] [ 4.799980] debug: unmapping init [mem 0xffffffffb5b14000-0xffffffffb5bfffff] [ 4.846890] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.854814] systemd[1]: Detected virtualization kvm. [ 4.855946] systemd[1]: Detected architecture x86-64. [ 4.863825] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.908992] systemd[1]: No hostname configured. [ 4.911522] systemd[1]: Set hostname to . [ 4.913773] random: systemd: uninitialized urandom read (16 bytes read) [ 4.916180] systemd[1]: Initializing machine ID from random generator. [ 5.129345] random: systemd: uninitialized urandom read (16 bytes read) [ 5.132526] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.147896] random: systemd: uninitialized urandom read (16 bytes read) [ 5.150863] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.160483] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.024595] device-mapper: uevent: version 1.0.3 [ 6.026512] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.183250] virtio_net virtio0 ens2: renamed from eth0 [ 7.216179] random: fast init done [ 7.541329] scsi host0: ata_piix [ 7.693203] scsi host1: ata_piix [ 7.694575] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.704623] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.263079] random: crng init done [ 12.264673] random: 7 urandom warning(s) missed due to ratelimiting [ 12.321825] dracut-initqueue[578]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.742481] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.588266] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.056329] SELinux: Disabled at runtime. [ 16.128405] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.141036] systemd[1]: Detected virtualization kvm. [ 16.143763] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.144771] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.147858] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.152478] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.162966] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.165679] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.178198] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.184900] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ 17.361227] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 17.857695] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.635816] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.640719] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.891326] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.000602] EDAC sbridge: Ver: 1.1.2 [ 21.901601] Key type dns_resolver registered [ 22.281976] NFS: Registering the id_resolver key type [ 22.283684] Key type id_resolver registered [ 22.286316] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg209-server login: [ 42.376666] libcfs: loading out-of-tree module taints kernel. [ 42.393055] alg: No test for adler32 (adler32-zlib) [ 43.144342] Key type ._llcrypt registered [ 43.145194] Key type .llcrypt registered [ 43.179713] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 49.591710] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 50.043442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.243233] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 50.481863] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 50.483523] LNet: Accept secure, port 988 [ 52.079130] Key type lgssc registered [ 52.569964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 58.823202] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 59.724766] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 63.042203] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 65.547304] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 68.177435] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 73.392023] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 77.723666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.752053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 78.834113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 78.843244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 78.883743] Lustre: lustre-MDT0000: new disk, initializing [ 78.907651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 78.915050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 80.312317] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 85.485297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 85.512337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.542101] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 85.549393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 85.552020] Lustre: Skipped 1 previous similar message [ 85.592231] Lustre: lustre-MDT0001: new disk, initializing [ 85.612599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 85.620432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 85.622826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 87.077951] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 92.452486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.486208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 92.492390] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 92.593187] Lustre: lustre-OST0000: new disk, initializing [ 92.594642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 92.611781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 94.109986] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 99.224304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.247066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 99.249660] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 99.286346] Lustre: lustre-OST0001: new disk, initializing [ 99.288174] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 99.302047] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 100.401241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 100.404922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 100.749892] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 106.204226] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 110.714629] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 117.195165] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing check_logdir /tmp/testlogs/ [ 118.752082] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing yml_node [ 120.281934] Lustre: DEBUG MARKER: Client: 2.15.7.3 [ 121.100458] Lustre: DEBUG MARKER: MDS: 2.15.7.3 [ 121.935530] Lustre: DEBUG MARKER: OSS: 2.15.7.3 [ 122.448075] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Oct 7 09:12:22 EDT 2025 [ 125.064289] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 125.562135] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 137.184389] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 137.187742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 137.193109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 141.279984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 141.280125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 141.284776] Lustre: Skipped 1 previous similar message [ 141.288698] Lustre: Skipped 3 previous similar messages [ 143.297965] Lustre: server umount lustre-MDT0000 complete [ 144.528711] LustreError: 6228:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759842766 with bad export cookie 1670722033334752353 [ 144.530262] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 144.531768] LustreError: 6228:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 146.399844] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 146.399919] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 146.400290] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 146.404613] Lustre: Skipped 2 previous similar messages [ 146.409481] LustreError: Skipped 1 previous similar message [ 149.407573] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 149.407704] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 149.410194] Lustre: Skipped 2 previous similar messages [ 149.414615] LustreError: Skipped 1 previous similar message [ 150.432045] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 150.438042] LustreError: Skipped 1 previous similar message [ 150.699622] Lustre: server umount lustre-MDT0001 complete [ 166.367106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 166.379644] Lustre: server umount lustre-OST0000 complete [ 173.780351] Lustre: server umount lustre-OST0001 complete [ 176.003727] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 178.728082] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 182.107575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 184.289339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 186.356542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.632572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.998508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.025330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.109209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 191.119762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 191.153351] Lustre: lustre-MDT0000: new disk, initializing [ 191.177798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.184479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.188091] Lustre: Skipped 1 previous similar message [ 192.658528] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 197.015340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 197.044227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.080267] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 197.176811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 197.181515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.185972] Lustre: Skipped 1 previous similar message [ 198.692321] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 202.634856] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 204.599635] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 206.880835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.906921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.910869] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 207.004344] Lustre: lustre-OST0000: new disk, initializing [ 207.006223] Lustre: Skipped 1 previous similar message [ 207.008744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 207.012332] Lustre: Skipped 2 previous similar messages [ 207.037826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.039480] Lustre: Skipped 1 previous similar message [ 208.562648] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 208.633088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.636584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.509383] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 214.064962] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 214.146808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 215.681954] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 215.760841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 219.106048] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 219.106086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 219.106725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 219.106729] Lustre: Skipped 2 previous similar messages [ 219.118971] Lustre: Skipped 1 previous similar message [ 222.570932] Lustre: server umount lustre-OST0000 complete [ 224.224444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 224.228567] Lustre: Skipped 1 previous similar message [ 229.343951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 229.346128] Lustre: Skipped 4 previous similar messages [ 231.199544] LustreError: 16395:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759842852 with bad export cookie 1670722033334753711 [ 231.203140] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 231.203426] LustreError: 16395:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 231.325107] Lustre: server umount lustre-MDT0001 complete [ 231.326434] Lustre: Skipped 1 previous similar message [ 232.933407] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 09:14:12 (1759842852) [ 235.214524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 235.381308] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 235.404056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.898301] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 239.973586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 241.804274] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 244.539716] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 246.466773] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 248.845390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.533853] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 253.152789] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 259.024768] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 265.698163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.699596] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 265.700129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.703567] Lustre: Skipped 2 previous similar messages [ 265.706597] LustreError: Skipped 1 previous similar message [ 268.171620] Lustre: server umount lustre-MDT0000 complete [ 269.428983] LustreError: 19500:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759842891 with bad export cookie 1670722033334754565 [ 269.432667] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 276.959195] Lustre: 21820:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759842892/real 1759842892] req@00000000c48ccf56 x1845328789696512/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759842898 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 284.140324] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 287.902395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 288.059777] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 288.079080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 288.082674] Lustre: Skipped 2 previous similar messages [ 289.689283] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 292.844635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 294.344215] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 297.209961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 298.683441] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 301.561193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 301.585600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 301.588451] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 301.639708] Lustre: lustre-OST0001: new disk, initializing [ 301.642088] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 303.074423] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 303.169155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 303.173411] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 307.474906] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 308.636451] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 318.431901] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 318.435281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.438491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.440809] Lustre: Skipped 2 previous similar messages [ 320.390613] Lustre: server umount lustre-MDT0000 complete [ 320.391833] Lustre: Skipped 2 previous similar messages [ 321.610895] LustreError: 26356:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759842943 with bad export cookie 1670722033334755412 [ 321.613343] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.616061] LustreError: 26356:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 331.231153] Lustre: 27062:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759842946/real 1759842946] req@0000000080b52c77 x1845328789720704/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759842952 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 337.765395] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 340.998547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 341.128088] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.503671] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 345.226045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 346.708071] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 349.421832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 350.896815] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 353.698339] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 353.769657] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 353.772242] Lustre: Skipped 6 previous similar messages [ 355.098135] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 363.527081] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 364.742314] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 364.744378] Lustre: Skipped 1 previous similar message [ 372.194906] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.199215] Lustre: Skipped 5 previous similar messages [ 378.742269] LustreError: 31488:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843000 with bad export cookie 1670722033334756658 [ 378.744842] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 378.748835] LustreError: 31488:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 385.503140] Lustre: 32095:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843001/real 1759843001] req@00000000644fa08a x1845328789740928/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759843007 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 385.513018] Lustre: 32095:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 385.529273] Lustre: server umount lustre-OST0000 complete [ 385.531401] Lustre: Skipped 5 previous similar messages [ 393.183188] Lustre: 32296:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843008/real 1759843008] req@00000000644fa08a x1845328789741376/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759843014 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 396.151097] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 09:16:56 (1759843016) [ 405.074773] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 408.264423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.392444] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 408.399840] LustreError: Skipped 1 previous similar message [ 409.828852] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 412.620844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.187100] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 417.084292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 418.638955] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 421.524119] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 423.038976] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 431.460190] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 432.595813] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 440.289753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.291831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.294438] Lustre: Skipped 4 previous similar messages [ 440.298075] Lustre: Skipped 15 previous similar messages [ 445.408616] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.415051] LustreError: Skipped 4 previous similar messages [ 446.220130] LustreError: 34311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843067 with bad export cookie 1670722033334757729 [ 446.222718] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.223793] LustreError: 34311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 453.599113] Lustre: 37882:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843069/real 1759843069] req@000000005f26eb8a x1845328789760896/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759843075 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 463.978839] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 09:18:03 (1759843083) [ 468.457254] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 471.384225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.791538] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 475.407968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.789730] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 479.482268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 481.037134] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 483.907341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 483.984967] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 483.987761] Lustre: Skipped 7 previous similar messages [ 485.451426] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 493.928759] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 506.336669] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 508.815330] LustreError: 39488:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843130 with bad export cookie 1670722033334758800 [ 508.818765] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 508.819528] LustreError: 39488:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 511.968132] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.972294] LustreError: Skipped 3 previous similar messages [ 523.231168] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 523.350130] Lustre: server umount lustre-MDT0001 complete [ 523.351973] Lustre: Skipped 6 previous similar messages [ 526.315037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.664813] Lustre: Evicted from MGS (at 192.168.202.109@tcp) after server handle changed from 0x172f98d663562d90 to 0x172f98d66356320c [ 541.669687] Lustre: MGC192.168.202.109@tcp: Connection restored to 192.168.202.109@tcp (at 0@lo) [ 541.804835] BUG: spinlock bad magic on CPU#0, lctl/43419 [ 541.807518] lock: 0xffff98e700cbdd00, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 541.810470] CPU: 0 PID: 43419 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 541.813803] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 541.816812] Call Trace: [ 541.818479] ? dump_stack+0xbb/0x10e [ 541.819794] ? spin_dump.cold.3+0x32/0x65 [ 541.820789] ? do_raw_spin_lock+0xce/0x190 [ 541.821991] ? _raw_spin_lock+0x21/0x30 [ 541.822957] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 541.824558] ? __seccomp_filter+0x4b/0x680 [ 541.826116] ? full_proxy_write+0x5e/0xa0 [ 541.827180] ? __vfs_write+0x1c/0x60 [ 541.828036] ? vfs_write+0xd8/0x2b0 [ 541.829232] ? ksys_write+0x66/0x120 [ 541.829980] ? __x64_sys_write+0x1e/0x30 [ 541.831437] ? do_syscall_64+0xc1/0x440 [ 541.833610] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 543.366113] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 546.484311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.165756] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 550.756374] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 550.756552] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 555.413311] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 557.328558] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 557.860444] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 560.506477] LustreError: 39488:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843182 with bad export cookie 1670722033334759948 [ 560.506487] LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 [ 560.511084] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.511923] LustreError: 39488:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 572.418901] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 09:19:52 (1759843192) [ 577.569145] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 581.094602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.763153] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 585.734987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.407768] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 590.253737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 591.770313] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 594.720503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 596.292718] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 604.595356] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 605.804995] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 605.806968] Lustre: Skipped 2 previous similar messages [ 616.928159] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 616.930833] LustreError: Skipped 1 previous similar message [ 616.932145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.935565] Lustre: Skipped 11 previous similar messages [ 616.937123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 616.938909] Lustre: Skipped 25 previous similar messages [ 627.679118] Lustre: 50827:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843243/real 1759843243] req@00000000727c06a8 x1845328789817152/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759843249 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 627.685991] Lustre: 50827:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 637.423425] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 09:20:57 (1759843257) [ 638.281837] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 638.862878] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 09:20:58 (1759843258) [ 642.580082] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 645.202866] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 648.529896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 650.563545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 652.872981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 652.896972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 652.989220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 653.026748] Lustre: lustre-MDT0000: new disk, initializing [ 653.054694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 654.530807] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 658.251591] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 660.486475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 660.512722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 660.515563] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 662.013625] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 665.796288] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 666.602367] Lustre: DEBUG MARKER: create test files [ 677.344359] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 688.607132] Lustre: 55514:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843304/real 1759843304] req@0000000095e89773 x1845328789824384/t0(0) o251->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759843310 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 688.614471] Lustre: 55514:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 693.709931] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 694.853278] Key type lgssc unregistered [ 694.995343] LNet: 56116:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.999364] LNet: Removed LNI 192.168.202.109@tcp [ 695.374563] Key type .llcrypt unregistered [ 695.376477] Key type ._llcrypt unregistered [ 706.284025] alg: No test for adler32 (adler32-zlib) [ 707.035309] Key type ._llcrypt registered [ 707.036324] Key type .llcrypt registered [ 707.075283] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 707.517869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 707.685594] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 707.783031] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 707.785772] LNet: Accept secure, port 988 [ 709.375148] Key type lgssc registered [ 709.717706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 712.345132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.507128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 714.935954] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 717.540779] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 719.836539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 719.943415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 721.486992] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 724.063001] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 729.123476] Lustre: lustre-OST0000: deleting orphan objects from 0x0:88 to 0x0:129 [ 729.149450] Lustre: 58956:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 729.152553] Lustre: 58956:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 729.154799] Lustre: 58956:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 729.156862] Lustre: 58956:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 729.159181] Lustre: 58956:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 729.161370] Lustre: 58956:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 732.647377] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 734.089687] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 739.296492] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 739.296494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 739.297080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 742.163115] Lustre: server umount lustre-OST0000 complete [ 749.535133] Lustre: 59969:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843365/real 1759843365] req@00000000d6b35818 x1845329478554624/t0(0) o251->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759843371 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 749.597713] Lustre: server umount lustre-MDT0000 complete [ 754.315572] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 755.294575] Key type lgssc unregistered [ 755.405412] LNet: 60572:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.408756] LNet: Removed LNI 192.168.202.109@tcp [ 755.689534] Key type .llcrypt unregistered [ 755.690616] Key type ._llcrypt unregistered [ 766.922908] alg: No test for adler32 (adler32-zlib) [ 767.674337] Key type ._llcrypt registered [ 767.675527] Key type .llcrypt registered [ 767.710646] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 768.032302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 768.269787] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 768.356669] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 768.359260] LNet: Accept secure, port 988 [ 769.943163] Key type lgssc registered [ 770.272704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 772.875325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 774.052175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.484487] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 777.937475] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 780.140689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 780.249099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 781.684605] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 784.209024] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 788.387179] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 788.404039] Lustre: 63507:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 788.406425] Lustre: 63507:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 788.408417] Lustre: 63507:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 788.410456] Lustre: 63507:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 788.412330] Lustre: 63507:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 788.414133] Lustre: 63507:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 792.518813] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 798.688284] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 798.688553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 798.694822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 800.659962] Lustre: server umount lustre-OST0000 complete [ 807.391117] Lustre: 64470:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843423/real 1759843423] req@000000008c2b5136 x1845329542517824/t0(0) o251->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759843429 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 807.454538] Lustre: server umount lustre-MDT0000 complete [ 812.351258] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 813.440951] Key type lgssc unregistered [ 813.557381] LNet: 65073:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.561464] LNet: Removed LNI 192.168.202.109@tcp [ 813.868078] Key type .llcrypt unregistered [ 813.869428] Key type ._llcrypt unregistered [ 819.973776] alg: No test for adler32 (adler32-zlib) [ 820.724315] Key type ._llcrypt registered [ 820.725301] Key type .llcrypt registered [ 820.760897] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 826.900413] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 827.243874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.499350] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 827.577898] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 827.579708] LNet: Accept secure, port 988 [ 829.159100] Key type lgssc registered [ 829.501090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 832.423854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 834.616865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 836.650238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 838.758803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 841.017689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 841.039290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 842.117595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 842.127805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 842.165188] Lustre: lustre-MDT0000: new disk, initializing [ 842.189398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.196773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 843.668658] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 847.855277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 847.875308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.907063] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 847.923903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 847.926045] Lustre: Skipped 1 previous similar message [ 847.963062] Lustre: lustre-MDT0001: new disk, initializing [ 847.999796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 848.011760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 848.016410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 849.547922] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 853.617541] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 855.673043] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 857.928969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 857.952056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 857.954709] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 858.042665] Lustre: lustre-OST0000: new disk, initializing [ 858.045787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 858.073242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.545623] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 860.980857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 860.984282] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 863.336153] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 864.935870] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 865.014694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 866.590740] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 866.664042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 871.393196] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 871.394163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.395720] LustreError: Skipped 1 previous similar message [ 871.400297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 871.402786] Lustre: Skipped 1 previous similar message [ 873.428633] Lustre: server umount lustre-OST0000 complete [ 876.513301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 876.514141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 876.516386] Lustre: Skipped 2 previous similar messages [ 876.519311] Lustre: Skipped 1 previous similar message [ 880.835638] Lustre: server umount lustre-MDT0000 complete [ 881.632840] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.636135] LustreError: Skipped 1 previous similar message [ 882.088085] LustreError: 68347:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843503 with bad export cookie 3193890610317854746 [ 882.090462] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 882.091628] LustreError: 68347:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 882.208558] Lustre: server umount lustre-MDT0001 complete [ 885.045781] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 09:25:04 (1759843504) [ 888.417838] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 892.792188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 901.232656] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 904.247235] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 907.739481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 909.890660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 911.927455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 913.950635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 916.175112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 916.195796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.278553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 916.288472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 916.326535] Lustre: lustre-MDT0000: new disk, initializing [ 916.351444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.357651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 917.863639] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 921.998453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 922.019693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.053690] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 922.068335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 922.070242] Lustre: Skipped 1 previous similar message [ 922.105328] Lustre: lustre-MDT0001: new disk, initializing [ 922.134882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 922.145258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 922.148953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 923.574868] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 927.441578] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 929.396311] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 931.707965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 931.734365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 931.737192] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 931.831215] Lustre: lustre-OST0000: new disk, initializing [ 931.833143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 931.859751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 933.303599] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 937.157527] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 937.460973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 937.463729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 938.726148] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 938.801575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 940.308198] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 940.403959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 942.562676] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 942.567063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.571664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 942.573740] Lustre: Skipped 1 previous similar message [ 943.208752] Lustre: server umount lustre-OST0000 complete [ 947.681597] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 947.681668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.684938] LustreError: Skipped 1 previous similar message [ 947.690261] Lustre: Skipped 1 previous similar message [ 947.694777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 947.696903] Lustre: Skipped 1 previous similar message [ 950.661014] Lustre: server umount lustre-MDT0000 complete [ 951.978300] LustreError: 76631:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843573 with bad export cookie 3193890610317855600 [ 951.980674] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 951.982599] LustreError: 76631:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 954.888611] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 09:26:14 (1759843574) [ 957.028711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.190681] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.213911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.633417] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 961.457097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.035700] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 965.502930] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 967.414619] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 969.675244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 971.251326] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 973.718206] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 974.950472] Lustre: 83924:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting c774a7d2-a484-4265-87fb-b852a3c71ec1 at adminstrative request [ 976.993174] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.998802] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 982.499522] Lustre: MGS: Client c774a7d2-a484-4265-87fb-b852a3c71ec1 (at 0@lo) reconnecting [ 982.502959] Lustre: Evicted from MGS (at 192.168.202.109@tcp) after server handle changed from 0x2c52fa968f738eb8 to 0x2c52fa968f73914a [ 982.508723] Lustre: MGC192.168.202.109@tcp: Connection restored to 192.168.202.109@tcp (at 0@lo) [ 1007.912963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1007.947814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1007.952394] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1008.025138] Lustre: lustre-OST0001: new disk, initializing [ 1008.028977] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1008.072035] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1008.076461] Lustre: Skipped 2 previous similar messages [ 1009.420407] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1013.461590] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1014.283988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1014.288298] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1019.367810] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.375174] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1019.375762] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1019.380576] Lustre: Skipped 2 previous similar messages [ 1020.674257] Lustre: server umount lustre-OST0001 complete [ 1020.676455] Lustre: Skipped 1 previous similar message [ 1023.455706] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1023.463715] LustreError: Skipped 1 previous similar message [ 1024.482401] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.498574] LustreError: Skipped 1 previous similar message [ 1033.184210] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1033.189255] LustreError: Skipped 1 previous similar message [ 1033.192995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.203262] Lustre: Skipped 2 previous similar messages [ 1033.210213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.213347] Lustre: Skipped 4 previous similar messages [ 1037.765982] Lustre: server umount lustre-MDT0000 complete [ 1037.778994] Lustre: Skipped 1 previous similar message [ 1039.841585] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.853479] LustreError: Skipped 1 previous similar message [ 1040.360693] LustreError: 81167:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843662 with bad export cookie 3193890610317857098 [ 1040.365798] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.369177] LustreError: 81167:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1044.131830] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 09:27:44 (1759843664) [ 1044.691673] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1045.335281] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 09:27:45 (1759843665) [ 1045.908588] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1046.548783] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 09:27:46 (1759843666) [ 1047.051224] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1047.659682] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 09:27:47 (1759843667) [ 1048.165687] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1049.759358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1050.096032] loop: module loaded [ 1050.099216] loop0: detected capacity change from 0 to 4294967296 [ 1050.110863] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1050.618827] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1051.718161] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1053.001596] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1055.911206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1057.207125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1058.420546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1059.678184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1061.812043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1061.831399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1061.833343] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1061.836801] LustreError: 88365:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1061.838882] LustreError: 88365:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1061.840919] LustreError: 88365:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1061.843491] LustreError: 88365:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1067.246179] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 1070.138430] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1073.318077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1075.528219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1077.547803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1079.568612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1081.796637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1081.820940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1081.895391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1081.904765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1081.937166] Lustre: lustre-MDT0000: new disk, initializing [ 1081.960210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.966124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1083.392699] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1087.431843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1087.458741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1087.571646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1088.936990] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1092.630114] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1094.518987] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1096.673209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1096.694560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1096.697466] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1098.138518] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1101.804260] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1103.190351] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1103.259191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1104.626939] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1104.708987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1107.435646] Lustre: server umount lustre-OST0000 complete [ 1107.437822] Lustre: Skipped 1 previous similar message [ 1113.057268] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1113.057611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.062601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1113.066852] Lustre: Skipped 2 previous similar messages [ 1113.071148] Lustre: Skipped 3 previous similar messages [ 1115.103139] Lustre: 94898:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843730/real 1759843730] req@000000006a1e17a3 x1845329604448704/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1759843736 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1115.111146] Lustre: 94898:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1116.410163] LustreError: 93305:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843738 with bad export cookie 3193890610317857679 [ 1116.413483] LustreError: 93305:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1116.413780] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1119.570242] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 09:28:59 (1759843739) [ 1121.850615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1122.003635] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1123.543884] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1126.559832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.248104] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1130.883088] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1132.789969] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1134.954464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1136.463225] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1139.021447] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1141.258667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1141.283919] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1141.288045] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1142.792277] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1143.485102] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1143.487382] Lustre: Skipped 1 previous similar message [ 1146.595547] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1156.399247] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1160.608920] Lustre: Failing over lustre-MDT0000 [ 1162.719708] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1162.722147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.725094] Lustre: Skipped 1 previous similar message [ 1162.726559] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.399113] Lustre: 66582:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843785/real 1759843785] req@00000000f8a040dd x1845329604471168/t0(0) o400->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759843792 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1170.405099] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1175.930442] LDISKFS-fs (dm-0): recovery complete [ 1175.931846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.545902] Lustre: Evicted from MGS (at 192.168.202.109@tcp) after server handle changed from 0x2c52fa968f7396d7 to 0x2c52fa968f7424a5 [ 1176.552272] Lustre: MGC192.168.202.109@tcp: Connection restored to (at 0@lo) [ 1176.660419] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1176.699628] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1176.701046] Lustre: 98228:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1178.194637] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1181.664096] Lustre: 95767:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1181.667931] Lustre: 95767:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 1 previous similar message [ 1181.672888] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1181.690815] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1182.007109] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1182.321268] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1182.326380] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 1182.639181] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1182.641368] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 1182.960825] Lustre: 100917:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1182.964234] Lustre: 100917:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 8 previous similar messages [ 1183.592695] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1183.594917] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 1183.911087] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1183.914786] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 1185.167716] Lustre: 100917:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1185.170700] Lustre: 100917:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 6 previous similar messages [ 1185.799911] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1185.802424] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 1186.119127] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1186.121254] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 1189.248880] Lustre: 100917:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1189.252432] Lustre: 100917:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages [ 1189.881034] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1189.883748] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 1190.199128] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1190.200872] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 1197.386413] Lustre: 100917:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1197.392955] Lustre: 100917:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 25 previous similar messages [ 1198.025177] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1198.028467] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 1198.343095] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1198.345368] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 1213.642507] Lustre: 100917:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1213.646928] Lustre: 100917:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 51 previous similar messages [ 1214.274218] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1214.280220] LustreError: 100917:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 51 previous similar messages [ 1214.599129] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1214.602694] LustreError: 100917:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 51 previous similar messages [ 1236.135123] Lustre: 100917:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1236.140532] Lustre: 100917:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1236.142912] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1236.144245] LustreError: 100917:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 1236.149115] Lustre: 100917:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@00000000dd1b8d1d x1845329597065792/t0(8589934771) o101->322c071d-43a8-4101-8f47-54f0cc5bbf82@192.168.202.9@tcp:18/0 lens 592/0 e 0 to 0 dl 1759843863 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 1236.157388] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1236.160722] Lustre: Skipped 2 previous similar messages [ 1236.161435] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1236.178300] Lustre: lustre-OST0000: deleting orphan objects from 0x0:78 to 0x0:577 [ 1236.178316] Lustre: lustre-OST0001: deleting orphan objects from 0x0:45 to 0x0:545 [ 1240.032290] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1240.036719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.044124] Lustre: Skipped 3 previous similar messages [ 1240.047806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1240.051754] Lustre: Skipped 2 previous similar messages [ 1245.551652] Lustre: server umount lustre-OST0000 complete [ 1245.553579] Lustre: Skipped 3 previous similar messages [ 1248.225572] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.228765] LustreError: Skipped 31 previous similar messages [ 1261.630571] LustreError: 101557:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759843883 with bad export cookie 3193890610317894821 [ 1261.633878] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1261.634269] LustreError: 101557:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1264.670794] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 09:31:24 (1759843884) [ 1267.319322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1267.324339] Lustre: *** cfs_fail_loc=197, val=0*** [ 1284.831553] LustreError: 102938:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 1: rc = -5 [ 1295.071369] LustreError: 102938:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 1301.224648] Lustre: *** cfs_fail_loc=197, val=0*** [ 1301.226666] Lustre: Skipped 65 previous similar messages [ 1324.767296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1324.770435] Lustre: Skipped 7 previous similar messages [ 1326.209859] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1328.812939] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1332.259729] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 09:32:32 (1759843952) [ 1346.636637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.810370] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.218182] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1352.701574] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1354.701731] LustreError: 104596:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1354.704796] LustreError: 104596:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1354.707687] LustreError: 104596:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1360.863313] Lustre: 105285:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759843976/real 1759843976] req@000000008301c5d4 x1845329604498752/t0(0) o251->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759843982 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1366.299699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1372.027619] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 1374.984415] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1378.234288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1380.385062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1382.312072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1384.427668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1386.623543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1386.645567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.718116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1386.720904] Lustre: Skipped 1 previous similar message [ 1386.729231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1386.730745] Lustre: Skipped 4 previous similar messages [ 1386.766117] Lustre: lustre-MDT0000: new disk, initializing [ 1386.767600] Lustre: Skipped 3 previous similar messages [ 1386.794365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1386.797031] Lustre: Skipped 3 previous similar messages [ 1388.129528] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1392.134294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1392.162064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1392.276356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1393.639941] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1397.289729] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1399.080174] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1401.199383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1401.222636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1401.225621] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1402.634172] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1406.283457] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1407.789694] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1411.940601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 1413.341395] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1413.421693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1416.674941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.675209] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1416.680021] Lustre: Skipped 5 previous similar messages [ 1416.682125] LustreError: Skipped 2 previous similar messages [ 1416.684147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1416.688163] Lustre: Skipped 9 previous similar messages [ 1420.183789] Lustre: server umount lustre-OST0000 complete [ 1420.185085] Lustre: Skipped 5 previous similar messages [ 1429.011483] LustreError: 109759:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844050 with bad export cookie 3193890610317905405 [ 1429.015507] LustreError: 109759:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1429.015614] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1432.015606] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 09:34:11 (1759844051) [ 1436.023957] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1437.018045] Key type lgssc unregistered [ 1437.140287] LNet: 113612:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.143320] LNet: Removed LNI 192.168.202.109@tcp [ 1437.452873] Key type .llcrypt unregistered [ 1437.454264] Key type ._llcrypt unregistered [ 1443.391746] alg: No test for adler32 (adler32-zlib) [ 1444.144306] Key type ._llcrypt registered [ 1444.145473] Key type .llcrypt registered [ 1444.179991] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 1450.312682] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1450.817428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1451.038461] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1451.127267] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1451.128837] LNet: Accept secure, port 988 [ 1452.719100] Key type lgssc registered [ 1453.086186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1455.984906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1458.140562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1460.100986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1462.158312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1464.015512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1466.156235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1466.178290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.257256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1467.266162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1467.302487] Lustre: lustre-MDT0000: new disk, initializing [ 1467.324517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1467.330670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1468.783127] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1472.882920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1472.910626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1472.914620] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1472.999557] Lustre: lustre-OST0000: new disk, initializing [ 1473.001216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1473.003638] Lustre: Skipped 1 previous similar message [ 1473.024325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.330151] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1478.314051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1478.334896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1478.337161] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1478.382276] Lustre: lustre-OST0001: new disk, initializing [ 1478.384808] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1478.410139] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1479.711384] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1487.328299] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.328823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1487.331774] Lustre: Skipped 1 previous similar message [ 1489.716228] Lustre: server umount lustre-MDT0000 complete [ 1491.807900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1493.320433] LustreError: 117958:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844115 with bad export cookie 15400787539589135248 [ 1493.322174] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1493.326131] LustreError: 117958:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1493.335992] Lustre: server umount lustre-OST0000 complete [ 1494.604395] Lustre: server umount lustre-OST0001 complete [ 1497.277590] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 09:35:17 (1759844117) [ 1499.867275] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1502.795818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1503.139690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1504.871526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1505.205872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1507.470605] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 09:35:27 (1759844127) [ 1511.736681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1512.401657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1513.212236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1513.922705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1514.234764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1514.871031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1520.361722] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 1523.262646] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1526.375592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1528.637554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1530.675766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1532.664802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1534.884813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1534.904294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.989436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1535.000464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1535.048534] Lustre: lustre-MDT0000: new disk, initializing [ 1535.073804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.079386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1536.423337] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1540.630652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1540.655251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1540.682730] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1540.697634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1540.700569] Lustre: Skipped 1 previous similar message [ 1540.737290] Lustre: lustre-MDT0001: new disk, initializing [ 1540.767145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1540.778530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1540.783130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1542.161490] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1545.960795] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1547.914677] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1550.220627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1550.244078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1550.246284] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1550.332986] Lustre: lustre-OST0000: new disk, initializing [ 1550.335479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1550.364551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.777109] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1552.760175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1552.763091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1555.964654] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1557.466291] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1557.534878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1558.955371] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1559.026806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1561.744870] Lustre: server umount lustre-OST0000 complete [ 1563.104349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1563.104841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1563.110116] Lustre: Skipped 1 previous similar message [ 1563.114194] Lustre: Skipped 1 previous similar message [ 1568.224056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1568.225843] Lustre: Skipped 2 previous similar messages [ 1569.247126] Lustre: 129297:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844184/real 1759844184] req@0000000019c86937 x1845330257665216/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1759844190 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1569.323877] Lustre: server umount lustre-MDT0000 complete [ 1570.271116] Lustre: 115112:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759844184/real 1759844184] req@00000000688423d4 x1845330257665344/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1759844191 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 1570.283444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.291885] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.299039] LustreError: Skipped 1 previous similar message [ 1570.796566] LustreError: 125612:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844192 with bad export cookie 15400787539589136438 [ 1570.801149] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1570.805618] LustreError: 125612:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1573.843835] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 09:36:33 (1759844193) [ 1576.421516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1576.606396] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1576.638964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.521533] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1581.765866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.257473] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1585.758085] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1587.693715] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1590.265617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1591.847445] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1594.327806] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1600.123105] Lustre: *** cfs_fail_loc=159, val=0*** [ 1607.303865] Lustre: lustre-MDT0000: Client 9b9a6d1a-3223-4bc6-bb4f-adba55fe8cc2 (at 192.168.202.9@tcp) reconnecting [ 1607.660907] Lustre: *** cfs_fail_loc=159, val=0*** [ 1607.662614] Lustre: Skipped 5 previous similar messages [ 1614.981512] Lustre: lustre-MDT0000: Client 9b9a6d1a-3223-4bc6-bb4f-adba55fe8cc2 (at 192.168.202.9@tcp) reconnecting [ 1617.378472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.379771] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1617.381278] Lustre: Skipped 1 previous similar message [ 1617.381437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1617.385419] LustreError: Skipped 1 previous similar message [ 1622.037077] Lustre: server umount lustre-OST0000 complete [ 1622.038286] Lustre: Skipped 1 previous similar message [ 1627.616272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1627.616886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1627.621531] Lustre: Skipped 1 previous similar message [ 1627.623107] Lustre: Skipped 1 previous similar message [ 1630.470059] LustreError: 132117:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844252 with bad export cookie 15400787539589137278 [ 1630.472570] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1630.472760] LustreError: 132117:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1634.989110] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1635.969308] Key type lgssc unregistered [ 1636.076338] LNet: 134020:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.079115] LNet: Removed LNI 192.168.202.109@tcp [ 1636.371886] Key type .llcrypt unregistered [ 1636.372823] Key type ._llcrypt unregistered [ 1644.884879] alg: No test for adler32 (adler32-zlib) [ 1645.636409] Key type ._llcrypt registered [ 1645.637280] Key type .llcrypt registered [ 1645.670956] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1646.110519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1646.209505] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1646.273427] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1646.274826] LNet: Accept secure, port 988 [ 1647.847065] Key type lgssc registered [ 1648.131111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1650.565484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.699974] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1651.713904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1652.983484] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1655.610660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1655.724960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1657.108145] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1659.444799] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1661.151297] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1663.220942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1663.368877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.981619] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1666.469284] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 1667.663596] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1677.275616] Lustre: *** cfs_fail_loc=159, val=0*** [ 1683.602539] Lustre: lustre-MDT0000: Client 8c77a2fe-76ca-41b6-a17b-da5adb92e9e0 (at 192.168.202.9@tcp) reconnecting [ 1684.360918] Lustre: *** cfs_fail_loc=159, val=0*** [ 1684.364197] Lustre: Skipped 1 previous similar message [ 1690.761840] Lustre: lustre-MDT0001: Client 8c77a2fe-76ca-41b6-a17b-da5adb92e9e0 (at 192.168.202.9@tcp) reconnecting [ 1691.172803] Lustre: *** cfs_fail_loc=159, val=0*** [ 1691.174477] Lustre: Skipped 1 previous similar message [ 1698.460314] Lustre: lustre-MDT0001: Client 8c77a2fe-76ca-41b6-a17b-da5adb92e9e0 (at 192.168.202.9@tcp) reconnecting [ 1700.743818] Lustre: *** cfs_fail_loc=159, val=0*** [ 1700.745712] Lustre: Skipped 3 previous similar messages [ 1708.189869] Lustre: lustre-MDT0000: Client 116f3406-ad96-4af9-ba45-0a82e182342a (at 192.168.202.9@tcp) reconnecting [ 1708.720642] Lustre: *** cfs_fail_loc=159, val=0*** [ 1708.722204] Lustre: Skipped 13 previous similar messages [ 1715.848272] Lustre: lustre-MDT0000: Client 116f3406-ad96-4af9-ba45-0a82e182342a (at 192.168.202.9@tcp) reconnecting [ 1721.826612] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1721.827750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.832926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1721.838102] Lustre: Skipped 1 previous similar message [ 1721.843665] Lustre: Skipped 1 previous similar message [ 1724.203044] Lustre: server umount lustre-OST0000 complete [ 1726.945031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.945932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1726.948052] Lustre: Skipped 1 previous similar message [ 1732.065272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1732.068799] Lustre: Skipped 2 previous similar messages [ 1737.183947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1737.185637] Lustre: Skipped 1 previous similar message [ 1740.255181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1740.294382] Lustre: server umount lustre-MDT0000 complete [ 1741.461861] LustreError: 135464:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844363 with bad export cookie 5009262782622209449 [ 1741.464566] LustreError: 135464:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1741.465905] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1741.597162] Lustre: server umount lustre-MDT0001 complete [ 1746.109669] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1747.092961] Key type lgssc unregistered [ 1747.203351] LNet: 139869:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1747.205731] LNet: Removed LNI 192.168.202.109@tcp [ 1747.501229] Key type .llcrypt unregistered [ 1747.502230] Key type ._llcrypt unregistered [ 1756.043981] alg: No test for adler32 (adler32-zlib) [ 1756.795382] Key type ._llcrypt registered [ 1756.796846] Key type .llcrypt registered [ 1756.833316] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1757.108963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1757.342885] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1757.410041] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1757.411850] LNet: Accept secure, port 988 [ 1758.983041] Key type lgssc registered [ 1759.259792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1761.605528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.730887] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.743062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1763.967115] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1766.487887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.592113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1767.802132] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1770.148563] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1771.897115] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1773.968137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1774.063525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.319141] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1777.629142] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1778.149523] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:129 [ 1782.246065] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 1786.785740] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1787.814524] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1787.816447] Lustre: Skipped 1 previous similar message [ 1792.481115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.485706] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1792.485916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1792.489220] Lustre: Skipped 1 previous similar message [ 1795.348372] Lustre: server umount lustre-OST0000 complete [ 1797.600882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.600959] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1797.602119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1797.603974] Lustre: Skipped 2 previous similar messages [ 1802.720166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.721668] Lustre: Skipped 2 previous similar messages [ 1802.823761] Lustre: server umount lustre-MDT0000 complete [ 1804.024912] LustreError: 141311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844425 with bad export cookie 17313947747940192158 [ 1804.027582] LustreError: 141311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1804.027693] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1804.142434] Lustre: server umount lustre-MDT0001 complete [ 1808.923618] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1810.207422] Key type lgssc unregistered [ 1810.340455] LNet: 145383:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1810.344549] LNet: Removed LNI 192.168.202.109@tcp [ 1810.743976] Key type .llcrypt unregistered [ 1810.745817] Key type ._llcrypt unregistered [ 1821.742412] alg: No test for adler32 (adler32-zlib) [ 1822.494267] Key type ._llcrypt registered [ 1822.495613] Key type .llcrypt registered [ 1822.531609] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1822.880468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1823.067128] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1823.141985] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1823.143577] LNet: Accept secure, port 988 [ 1824.719094] Key type lgssc registered [ 1825.254167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1828.343313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.472869] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1829.488641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.737615] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1833.347974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.460764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1834.743762] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1837.014267] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1838.810340] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1840.851607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1840.956104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.209389] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1844.418206] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1845.092133] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:161 [ 1849.188450] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 1852.829640] Lustre: *** cfs_fail_loc=159, val=0*** [ 1860.229552] Lustre: lustre-MDT0000: Client 2e46e8e8-0f0d-4fce-a312-9ef4d7703682 (at 192.168.202.9@tcp) reconnecting [ 1864.673358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.673481] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1864.674040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.676570] Lustre: Skipped 1 previous similar message [ 1867.345712] Lustre: server umount lustre-OST0000 complete [ 1869.279732] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1869.281730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.284825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1869.287033] Lustre: Skipped 1 previous similar message [ 1874.628296] Lustre: server umount lustre-MDT0000 complete [ 1874.911942] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1874.917802] LustreError: Skipped 1 previous similar message [ 1875.804965] LustreError: 146826:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844497 with bad export cookie 1405400845760788286 [ 1875.807467] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1875.807875] LustreError: 146826:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1875.946173] Lustre: server umount lustre-MDT0001 complete [ 1880.482577] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1881.398092] Key type lgssc unregistered [ 1881.510290] LNet: 150605:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.512590] LNet: Removed LNI 192.168.202.109@tcp [ 1881.799182] Key type .llcrypt unregistered [ 1881.800675] Key type ._llcrypt unregistered [ 1890.118329] alg: No test for adler32 (adler32-zlib) [ 1890.869341] Key type ._llcrypt registered [ 1890.870205] Key type .llcrypt registered [ 1890.902219] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1891.208199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1891.402418] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 1891.471937] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1891.473272] LNet: Accept secure, port 988 [ 1893.055068] Key type lgssc registered [ 1893.409334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1895.834071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.963402] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1896.978274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.212518] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1901.458213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.565491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1902.832892] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1905.203081] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1907.031447] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1909.217689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1909.328930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.594329] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 1912.421396] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 1912.783543] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1916.517184] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 1931.368848] Lustre: lustre-OST0000: evicting eb288c4b-46c5-43ef-8203-1ebad39e1b23 (at 192.168.202.9@tcp) by administrative request [ 1931.371321] Lustre: Skipped 1 previous similar message [ 1957.857856] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1957.857926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.861366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1957.863686] Lustre: Skipped 1 previous similar message [ 1957.868139] Lustre: Skipped 1 previous similar message [ 1961.425253] Lustre: server umount lustre-OST0000 complete [ 1962.976334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.976363] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1962.976832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.979365] Lustre: Skipped 1 previous similar message [ 1968.096165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1968.098157] Lustre: Skipped 2 previous similar messages [ 1968.774718] Lustre: server umount lustre-MDT0000 complete [ 1970.014042] LustreError: 152048:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844591 with bad export cookie 3047305166752902200 [ 1970.016677] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1970.017972] LustreError: 152048:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1970.138156] Lustre: server umount lustre-MDT0001 complete [ 1974.860290] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1975.875198] Key type lgssc unregistered [ 1975.976456] LNet: 156259:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.978783] LNet: Removed LNI 192.168.202.109@tcp [ 1976.264685] Key type .llcrypt unregistered [ 1976.266596] Key type ._llcrypt unregistered [ 2003.740379] alg: No test for adler32 (adler32-zlib) [ 2004.492296] Key type ._llcrypt registered [ 2004.493332] Key type .llcrypt registered [ 2004.552411] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2005.027217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2005.278356] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2005.360687] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2005.362230] LNet: Accept secure, port 988 [ 2006.943242] Key type lgssc registered [ 2007.549962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2011.320632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.490723] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2012.511414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.947362] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2017.761868] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.046205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.212935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2020.075792] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2023.513757] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2025.925673] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2028.821701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2028.963877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.732624] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2034.039410] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2034.088552] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 2042.281897] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 2049.504035] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2049.506621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.510979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2050.018615] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2050.022842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2050.029938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2052.578675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2052.582503] Lustre: Skipped 1 previous similar message [ 2053.681561] Lustre: server umount lustre-OST0000 complete [ 2057.697451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.699104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.703788] Lustre: Skipped 1 previous similar message [ 2057.710799] Lustre: Skipped 1 previous similar message [ 2061.456068] Lustre: server umount lustre-MDT0000 complete [ 2062.816981] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.829842] LustreError: Skipped 1 previous similar message [ 2063.290570] LustreError: 160748:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844685 with bad export cookie 14783527867300162016 [ 2063.296994] LustreError: 160748:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2063.297049] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2063.450632] Lustre: server umount lustre-MDT0001 complete [ 2069.609814] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2070.822403] Key type lgssc unregistered [ 2070.976391] LNet: 162475:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.980595] LNet: Removed LNI 192.168.202.109@tcp [ 2071.392663] Key type .llcrypt unregistered [ 2071.394064] Key type ._llcrypt unregistered [ 2083.256983] alg: No test for adler32 (adler32-zlib) [ 2084.010324] Key type ._llcrypt registered [ 2084.011214] Key type .llcrypt registered [ 2084.049127] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2084.440019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2084.576326] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2084.645802] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2084.647600] LNet: Accept secure, port 988 [ 2086.223145] Key type lgssc registered [ 2086.774338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2089.933958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2093.157489] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 09:45:12 (1759844712) [ 2100.352948] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 2102.875437] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2105.706447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2108.604494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2111.613871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2114.653678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2117.754290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2117.790950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.879869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2118.889128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2118.925458] Lustre: lustre-MDT0000: new disk, initializing [ 2118.947396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.953164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2120.625590] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2124.766384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2124.787728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2124.814853] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2124.829357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2124.830906] Lustre: Skipped 1 previous similar message [ 2124.863616] Lustre: lustre-MDT0001: new disk, initializing [ 2124.892331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2124.903086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2124.906543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2126.505677] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2131.163481] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2133.867831] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2136.517752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2136.535723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2136.538109] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2136.620869] Lustre: lustre-OST0000: new disk, initializing [ 2136.622853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2136.645864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.785644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2136.789550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2138.440867] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2142.845862] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2147.299240] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2147.299331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.304796] LustreError: Skipped 1 previous similar message [ 2147.312507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2147.318733] Lustre: Skipped 1 previous similar message [ 2152.420061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2152.424505] Lustre: Skipped 1 previous similar message [ 2153.081230] Lustre: server umount lustre-OST0000 complete [ 2155.488337] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2155.492834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2155.499302] Lustre: Skipped 1 previous similar message [ 2155.502285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2157.536870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2157.537869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.707945] Lustre: server umount lustre-MDT0000 complete [ 2162.325615] LustreError: 166865:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844784 with bad export cookie 9754647740604854860 [ 2162.330048] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2162.332068] LustreError: 166865:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2162.481905] Lustre: server umount lustre-MDT0001 complete [ 2168.049857] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2169.357577] Key type lgssc unregistered [ 2169.506465] LNet: 170872:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.511329] LNet: Removed LNI 192.168.202.109@tcp [ 2169.973770] Key type .llcrypt unregistered [ 2169.975512] Key type ._llcrypt unregistered [ 2181.492066] alg: No test for adler32 (adler32-zlib) [ 2182.245358] Key type ._llcrypt registered [ 2182.246447] Key type .llcrypt registered [ 2182.284559] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2182.732621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.831106] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2182.896764] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2182.898360] LNet: Accept secure, port 988 [ 2184.479119] Key type lgssc registered [ 2184.829000] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2188.230615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.400429] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.414898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.164472] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2194.913479] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.375431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.553955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2197.489927] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2201.054755] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2203.715672] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2207.012670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2207.166189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.131935] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2212.752254] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2219.495118] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2225.442478] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2225.449713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.454789] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2225.457791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2225.462423] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.109@tcp (at 0@lo) [ 2225.462960] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2225.814838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2225.820486] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 2226.131427] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2226.133243] Lustre: Skipped 1 previous similar message [ 2226.137139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.140016] Lustre: Skipped 1 previous similar message [ 2226.141114] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2226.142643] Lustre: Skipped 1 previous similar message [ 2226.143778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2226.147163] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.109@tcp (at 0@lo) [ 2226.147316] Lustre: lustre-OST0000: deleting orphan objects from 0x0:181 to 0x0:225 [ 2226.148932] Lustre: Skipped 1 previous similar message [ 2226.489432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2226.498320] Lustre: lustre-OST0000: deleting orphan objects from 0x0:181 to 0x0:257 [ 2226.929373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2226.938834] Lustre: lustre-OST0000: deleting orphan objects from 0x0:181 to 0x0:289 [ 2227.326820] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2227.331153] Lustre: Skipped 2 previous similar messages [ 2227.339688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.348850] Lustre: Skipped 2 previous similar messages [ 2227.352456] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2227.356095] Lustre: Skipped 2 previous similar messages [ 2227.358692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2227.367080] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.109@tcp (at 0@lo) [ 2227.367422] Lustre: lustre-OST0000: deleting orphan objects from 0x0:181 to 0x0:321 [ 2227.371234] Lustre: Skipped 2 previous similar messages [ 2227.849181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2227.855097] Lustre: lustre-OST0000: deleting orphan objects from 0x0:181 to 0x0:353 [ 2228.264750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2228.274186] Lustre: lustre-OST0000: deleting orphan objects from 0x0:181 to 0x0:385 [ 2228.699361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2228.708922] Lustre: lustre-OST0000: deleting orphan objects from 0x0:181 to 0x0:417 [ 2229.135220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2229.142896] Lustre: lustre-OST0000: deleting orphan objects from 0x0:181 to 0x0:449 [ 2234.336425] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2234.341825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.349162] Lustre: Skipped 4 previous similar messages [ 2234.352428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2234.852521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2238.395622] Lustre: server umount lustre-OST0000 complete [ 2241.503670] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2241.505489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.508140] Lustre: Skipped 1 previous similar message [ 2241.509458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2245.088670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2245.093267] Lustre: Skipped 1 previous similar message [ 2246.359374] Lustre: server umount lustre-MDT0000 complete [ 2247.861395] LustreError: 173119:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759844869 with bad export cookie 17313688284356328429 [ 2247.865380] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.867886] LustreError: 173119:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2248.005376] Lustre: server umount lustre-MDT0001 complete [ 2253.483423] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2254.740397] Key type lgssc unregistered [ 2254.845401] LNet: 176809:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2254.848784] LNet: Removed LNI 192.168.202.109@tcp [ 2255.236987] Key type .llcrypt unregistered [ 2255.238240] Key type ._llcrypt unregistered [ 2265.251485] alg: No test for adler32 (adler32-zlib) [ 2266.003292] Key type ._llcrypt registered [ 2266.004195] Key type .llcrypt registered [ 2266.042240] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2266.369631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2266.581483] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2266.655789] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2266.657385] LNet: Accept secure, port 988 [ 2268.239108] Key type lgssc registered [ 2268.648122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2271.746083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.915997] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.929756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2274.507295] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2278.050130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.243836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2279.760417] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2283.355464] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2286.135297] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2289.237473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2289.404394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.350991] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2295.165764] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2302.766564] Lustre: lustre-OST0000: deleting orphan objects from 0x0:181 to 0x0:481 [ 2307.966144] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2308.756801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.765839] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2308.773808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2308.779863] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.109@tcp (at 0@lo) [ 2308.779926] LustreError: 178288:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2310.786324] Lustre: lustre-OST0000: Not available for connect from 192.168.202.9@tcp (stopping) [ 2313.187128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.189071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2313.196259] Lustre: Skipped 1 previous similar message [ 2315.835246] Lustre: server umount lustre-OST0000 complete [ 2328.863141] LustreError: 178288:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 2328.867424] LustreError: 178288:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2344.951480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2345.110238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.149882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2346.155662] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2346.155778] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2346.161838] Lustre: lustre-OST0000: deleting orphan objects from 0x0:181 to 0x0:513 [ 2346.167450] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:66 to 0x280000400:97 [ 2346.169797] Lustre: Skipped 1 previous similar message [ 2346.673698] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2349.904960] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2365.476350] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 14 sec [ 2368.266471] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9064d9006800.ost_server_uuid 40 [ 2368.893593] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9064d9006800.ost_server_uuid in FULL state after 0 sec [ 2372.065140] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2372.065261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.067139] LustreError: Skipped 1 previous similar message [ 2372.071741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.075101] Lustre: Skipped 2 previous similar messages [ 2376.277970] Lustre: server umount lustre-OST0000 complete [ 2380.768145] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2380.772912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2380.780312] Lustre: Skipped 1 previous similar message [ 2380.783662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2387.424122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2387.425644] Lustre: Skipped 3 previous similar messages [ 2392.031200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2392.102084] Lustre: server umount lustre-MDT0000 complete [ 2392.544851] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.552143] LustreError: Skipped 1 previous similar message [ 2393.812624] LustreError: 178251:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845015 with bad export cookie 12936486624629671047 [ 2393.816297] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2393.822676] LustreError: 178251:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2398.684269] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2399.562605] Key type lgssc unregistered [ 2399.682354] LNet: 183608:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.684787] LNet: Removed LNI 192.168.202.109@tcp [ 2399.954785] Key type .llcrypt unregistered [ 2399.956079] Key type ._llcrypt unregistered [ 2413.297877] alg: No test for adler32 (adler32-zlib) [ 2414.049473] Key type ._llcrypt registered [ 2414.051044] Key type .llcrypt registered [ 2414.088148] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2414.486512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2414.618277] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2414.685300] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2414.687203] LNet: Accept secure, port 988 [ 2416.263138] Key type lgssc registered [ 2416.842717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2421.071976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.229390] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.244720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.873667] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2427.361322] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.802271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.019197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2429.956161] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2434.026966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2434.209580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2435.961542] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2437.292497] Lustre: lustre-OST0000: deleting orphan objects from 0x0:515 to 0x0:545 [ 2439.533589] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2439.563641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2439.566275] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2439.624287] Lustre: lustre-OST0001: new disk, initializing [ 2439.626637] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2439.661436] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2441.608344] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2444.299955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2444.307119] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2448.261667] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2451.255470] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2468.832462] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2468.837628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.846113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2469.858190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2469.861911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2469.864984] Lustre: Skipped 1 previous similar message [ 2469.872279] Lustre: Skipped 1 previous similar message [ 2471.445210] Lustre: server umount lustre-MDT0000 complete [ 2473.257922] LustreError: 185363:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845094 with bad export cookie 8434369923901776121 [ 2473.263221] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2473.264354] LustreError: 185363:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2473.436767] Lustre: server umount lustre-MDT0001 complete [ 2480.607116] Lustre: 189170:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759845096/real 1759845096] req@0000000087e80d95 x1845331268513024/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759845102 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2480.646803] Lustre: server umount lustre-OST0000 complete [ 2482.144143] Lustre: 184647:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759845096/real 1759845096] req@0000000067613212 x1845331268512896/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759845103 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 2482.158384] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.166240] Lustre: Skipped 1 previous similar message [ 2484.066729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2484.109185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2484.595971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2484.658019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2485.146733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2485.190916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2485.713454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2485.760350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2491.963544] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2496.231822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.484481] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.512043] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.004657] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2501.601697] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.720306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.874239] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2507.853266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2508.011647] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2508.015448] Lustre: Skipped 1 previous similar message [ 2509.643667] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2512.100284] LustreError: 137-5: mylustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.102904] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:548 to 0x0:577 [ 2513.322965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2515.126888] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2522.628341] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2524.341164] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2536.291117] Lustre: server umount mylustre-MDT0000 complete [ 2536.292865] Lustre: Skipped 1 previous similar message [ 2537.443061] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.449688] LustreError: 137-5: mylustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.450141] Lustre: Skipped 2 previous similar messages [ 2537.463569] LustreError: Skipped 3 previous similar messages [ 2537.901753] LustreError: 194020:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845159 with bad export cookie 8434369923901778830 [ 2537.907926] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2537.911530] LustreError: 194020:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2547.679102] Lustre: 194532:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759845163/real 1759845163] req@00000000c1cf911b x1845331268537664/t0(0) o39->mylustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759845169 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2547.700487] Lustre: server umount mylustre-OST0000 complete [ 2547.702166] Lustre: Skipped 1 previous similar message [ 2554.335247] Lustre: 194733:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759845170/real 1759845170] req@00000000e68c8c12 x1845331268538176/t0(0) o39->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759845176 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2555.770739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2555.807360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2556.269444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2556.313691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2556.815366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2556.859525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2557.344449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2557.397283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2563.157284] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2567.280529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.489886] LustreError: 137-5: tfs-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.509258] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2567.511298] Lustre: Skipped 1 previous similar message [ 2569.392925] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2573.232995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.272629] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2578.831881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2580.916791] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2583.082755] Lustre: tfs-OST0000: deleting orphan objects from 0x0:548 to 0x0:609 [ 2585.709757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2587.649909] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 2587.694837] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2591.830852] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2593.486485] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2604.003630] Lustre: tfs-MDT0000-lwp-OST0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.006844] LustreError: 11-0: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2604.008827] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 2604.012643] Lustre: Skipped 4 previous similar messages [ 2606.922596] Lustre: server umount tfs-MDT0000 complete [ 2606.924063] Lustre: Skipped 1 previous similar message [ 2607.011219] LustreError: 137-5: tfs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.016386] LustreError: Skipped 5 previous similar messages [ 2608.221743] LustreError: 199180:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845229 with bad export cookie 8434369923901780440 [ 2608.224787] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.224903] LustreError: 199180:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2615.263166] Lustre: 199835:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759845231/real 1759845231] req@00000000fe304d20 x1845331268557312/t0(0) o39->tfs-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759845237 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2623.834740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2623.865650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2624.232220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2624.256749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2624.570776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2624.590137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2624.908322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2624.933239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2630.764772] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2634.684034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.906087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.908992] Lustre: Skipped 3 previous similar messages [ 2636.444804] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2640.072723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.829081] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2644.785278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2646.328966] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2649.262611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2650.415865] Lustre: lustre-OST0000: deleting orphan objects from 0x0:548 to 0x0:641 [ 2650.418528] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 2651.217460] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2654.687262] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2655.878134] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2663.382103] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 09:54:43 (1759845283) [ 2676.195558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.200238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2676.203037] Lustre: Skipped 2 previous similar messages [ 2676.206670] Lustre: Skipped 4 previous similar messages [ 2676.211717] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2681.313610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2681.317045] Lustre: Skipped 4 previous similar messages [ 2681.990957] Lustre: server umount lustre-MDT0000 complete [ 2681.992164] Lustre: Skipped 3 previous similar messages [ 2683.208845] LustreError: 204491:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845304 with bad export cookie 8434369923901782071 [ 2683.212035] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2683.214870] LustreError: 204491:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2686.432076] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.436445] LustreError: Skipped 3 previous similar messages [ 2705.375110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2715.357843] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 2718.350770] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2722.075255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2724.340665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2726.187151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2728.617993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2731.847296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2731.878856] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 2731.878856] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2731.878856] [ 2731.891484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2731.991427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2732.005911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2732.056870] Lustre: lustre-MDT0000: new disk, initializing [ 2732.081799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.084957] Lustre: Skipped 3 previous similar messages [ 2732.091114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2733.644654] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2738.931715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2738.959771] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 2738.959771] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2738.959771] [ 2738.973903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2739.059375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2739.063770] Lustre: Skipped 1 previous similar message [ 2739.109972] Lustre: lustre-MDT0001: new disk, initializing [ 2739.177736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2739.181638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2741.309509] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2746.684641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2746.716286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2746.719961] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2746.838476] Lustre: lustre-OST0000: new disk, initializing [ 2746.841946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2748.456178] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2748.734172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2748.742125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2753.510291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2753.551643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2753.556260] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2753.638194] Lustre: lustre-OST0001: new disk, initializing [ 2753.641818] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2755.551327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2755.557510] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2755.709425] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2761.368342] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 09:56:21 (1759845381) [ 2766.218816] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 09:56:25 (1759845385) [ 2779.617386] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2779.617498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2779.624028] LustreError: Skipped 1 previous similar message [ 2779.624804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2779.631979] Lustre: Skipped 4 previous similar messages [ 2779.634437] Lustre: Skipped 4 previous similar messages [ 2783.860589] Lustre: server umount lustre-OST0000 complete [ 2783.863409] Lustre: Skipped 3 previous similar messages [ 2793.406493] LustreError: 207758:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845415 with bad export cookie 8434369923901783590 [ 2793.410021] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2793.410707] LustreError: 207758:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2796.513910] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2796.518557] Lustre: Skipped 8 previous similar messages [ 2804.673693] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2818.301880] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 2826.229883] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2827.257083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2829.984973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2832.109683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2834.079691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2836.299794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2838.654973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2838.682676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.779926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2838.782331] Lustre: Skipped 1 previous similar message [ 2838.794844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2838.837721] Lustre: lustre-MDT0000: new disk, initializing [ 2838.873917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2840.267152] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2844.430636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2844.451766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.585187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2846.023287] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2849.860808] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2851.767892] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2854.096783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2854.126211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2854.128461] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2856.248570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2856.254092] Lustre: Skipped 1 previous similar message [ 2856.256682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2856.918078] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2864.072641] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2871.776403] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2871.778060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2871.782030] LustreError: Skipped 1 previous similar message [ 2871.795844] Lustre: Skipped 4 previous similar messages [ 2871.800206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2871.806734] Lustre: Skipped 2 previous similar messages [ 2874.035084] Lustre: server umount lustre-OST0000 complete [ 2874.036583] Lustre: Skipped 3 previous similar messages [ 2887.056074] LustreError: 214940:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845508 with bad export cookie 8434369923901785018 [ 2887.058674] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2887.062590] LustreError: 214940:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2904.366516] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2908.393181] Key type lgssc unregistered [ 2908.925511] LNet: 218804:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2908.940340] LNet: Removed LNI 192.168.202.109@tcp [ 2910.257233] Key type .llcrypt unregistered [ 2910.266190] Key type ._llcrypt unregistered [ 2931.892504] alg: No test for adler32 (adler32-zlib) [ 2932.645472] Key type ._llcrypt registered [ 2932.647143] Key type .llcrypt registered [ 2932.717548] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2933.180761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2933.432667] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 2933.544976] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2933.548616] LNet: Accept secure, port 988 [ 2935.151150] Key type lgssc registered [ 2935.734872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2940.258748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.492111] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2941.526937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2944.087198] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2946.532339] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2948.757153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.925486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2950.901834] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2954.518918] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2957.634429] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2960.929698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2961.098647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.222384] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 2966.994595] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2967.502027] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 2969.572433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.572460] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2969.579529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2970.606655] Lustre: server umount lustre-OST0000 complete [ 2973.094478] Lustre: server umount lustre-MDT0000 complete [ 2974.690706] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2974.695155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.705987] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.579797] LustreError: 220294:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845597 with bad export cookie 15482572637186196088 [ 2975.585272] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2975.586081] LustreError: 220294:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2981.867044] Lustre: server umount lustre-MDT0001 complete [ 2988.086913] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2989.764920] Key type lgssc unregistered [ 2989.950875] LNet: 224016:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2989.955682] LNet: Removed LNI 192.168.202.109@tcp [ 2990.425711] Key type .llcrypt unregistered [ 2990.427224] Key type ._llcrypt unregistered [ 2999.413664] alg: No test for adler32 (adler32-zlib) [ 3000.165403] Key type ._llcrypt registered [ 3000.166916] Key type .llcrypt registered [ 3000.242653] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3000.652545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3000.953389] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3001.056910] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3001.059747] LNet: Accept secure, port 988 [ 3002.663180] Key type lgssc registered [ 3003.195545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3007.718863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3024.223316] LustreError: 225256:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 1: rc = -5 [ 3055.967272] LustreError: 225256:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 3085.664987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.561198] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3090.687453] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3091.626718] Lustre: server umount lustre-OST0000 complete [ 3099.954308] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3101.520142] Key type lgssc unregistered [ 3101.687956] LNet: 226912:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3101.694454] LNet: Removed LNI 192.168.202.109@tcp [ 3102.145383] Key type .llcrypt unregistered [ 3102.148832] Key type ._llcrypt unregistered [ 3124.868173] alg: No test for adler32 (adler32-zlib) [ 3125.626393] Key type ._llcrypt registered [ 3125.628140] Key type .llcrypt registered [ 3125.717996] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3126.327852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3126.506807] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3126.645984] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3126.650928] LNet: Accept secure, port 988 [ 3128.271126] Key type lgssc registered [ 3129.143262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3136.706914] loop0: detected capacity change from 0 to 204800000 [ 3136.849378] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3136.855931] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3137.034015] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3137.041405] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3137.071844] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3137.098966] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3137.121835] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3137.168373] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3137.203562] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3137.249820] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3137.949945] loop0: detected capacity change from 0 to 204800000 [ 3138.069945] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3138.444334] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3138.995678] loop0: detected capacity change from 0 to 204800000 [ 3139.239279] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3139.732777] loop0: detected capacity change from 0 to 204800000 [ 3139.957479] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3140.622960] loop0: detected capacity change from 0 to 204800000 [ 3140.645678] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3143.001368] loop0: detected capacity change from 0 to 204800000 [ 3143.020548] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3145.122391] loop0: detected capacity change from 0 to 204800000 [ 3145.141152] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3147.383878] loop0: detected capacity change from 0 to 204800000 [ 3147.404944] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3149.616972] loop0: detected capacity change from 0 to 204800000 [ 3149.673504] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3149.717063] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.761610] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3149.768620] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3149.775412] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3149.781675] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3149.789989] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3149.797459] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3149.804135] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3149.811360] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3149.817919] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3149.824529] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3149.829973] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3149.836583] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3149.842586] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3149.847858] Lustre: 230369:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3159.519398] Lustre: 230648:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759845775/real 1759845775] req@0000000014e84610 x1845332015054976/t0(0) o251->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759845781 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3159.539527] Lustre: server umount MGS complete [ 3160.116365] loop0: detected capacity change from 0 to 204800000 [ 3160.217781] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.435708] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.502968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3160.542964] LustreError: 230702:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 3161.236314] loop1: detected capacity change from 0 to 204800000 [ 3161.325809] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3161.378197] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.420803] Lustre: 230817:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3161.426033] Lustre: 230817:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3161.430983] Lustre: 230817:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3161.435591] Lustre: 230817:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3161.440589] Lustre: 230817:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3161.447198] Lustre: 230817:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3161.453376] Lustre: 230817:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3161.471928] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3161.598677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3161.641734] LustreError: 230817:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0001: Aborting recovery for device [ 3162.315494] loop2: detected capacity change from 0 to 204800000 [ 3162.426328] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3162.492975] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3162.496901] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 3162.522778] Lustre: 230900:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3162.527216] Lustre: 230900:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3162.533959] Lustre: 230900:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3162.539811] Lustre: 230900:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3162.546787] Lustre: 230900:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3162.554685] Lustre: 230900:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3162.560665] Lustre: 230900:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3162.658919] Lustre: Found index 0 for lustre-OST0000, updating log [ 3162.756771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3162.760377] LustreError: 230760:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 3162.776484] LustreError: 230900:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 3163.401762] loop3: detected capacity change from 0 to 204800000 [ 3163.516625] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3163.546272] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3163.550567] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 3163.574562] Lustre: 230988:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3163.581208] Lustre: 230988:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3163.588602] Lustre: 230988:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3163.597965] Lustre: 230988:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3163.609803] Lustre: 230988:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3163.615945] Lustre: 230988:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3163.621742] Lustre: 230988:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3163.738920] LustreError: 230988:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 3165.732319] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 3165.734514] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 3166.760322] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 3166.765945] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 3170.267959] LustreError: 231178:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 2222us: rc = -2 [ 3170.780132] LustreError: 230728:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1235us: rc = -2 [ 3170.789717] LustreError: 230728:0:(mdt_identity.c:112:mdt_identity_do_upcall()) Skipped 42 previous similar messages [ 3182.049978] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3182.050505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3182.050646] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3182.055395] Lustre: Skipped 2 previous similar messages [ 3182.057816] Lustre: Skipped 1 previous similar message [ 3187.168426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3187.175159] Lustre: Skipped 5 previous similar messages [ 3192.290845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3192.294340] Lustre: Skipped 3 previous similar messages [ 3196.383153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3196.427877] Lustre: server umount lustre-MDT0000 complete [ 3196.854595] LustreError: 230710:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845818 with bad export cookie 3936041556640952535 [ 3196.856352] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3196.858475] LustreError: 230710:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3197.407594] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.407744] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.408147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3197.411152] LustreError: Skipped 1 previous similar message [ 3197.420675] Lustre: Skipped 2 previous similar messages [ 3201.439577] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3201.439631] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3201.442443] Lustre: Skipped 2 previous similar messages [ 3201.447445] LustreError: Skipped 1 previous similar message [ 3207.647755] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3207.653529] LustreError: Skipped 3 previous similar messages [ 3211.231133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3211.318668] Lustre: server umount lustre-MDT0001 complete [ 3217.769938] Lustre: server umount lustre-OST0000 complete [ 3224.217635] Lustre: server umount lustre-OST0001 complete [ 3226.835722] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 10:04:06 (1759845846) [ 3235.839550] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 3239.024102] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3242.525620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3244.541666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3246.418128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3248.555671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3250.763650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3250.791203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.876571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3250.890455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3250.930393] Lustre: lustre-MDT0000: new disk, initializing [ 3250.959540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3250.962235] Lustre: Skipped 1 previous similar message [ 3250.970113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3252.431263] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3256.586616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3256.606968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.634867] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3256.651610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3256.653413] Lustre: Skipped 1 previous similar message [ 3256.689616] Lustre: lustre-MDT0001: new disk, initializing [ 3256.720175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3256.733892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3256.737239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3258.092324] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3261.973442] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3264.169452] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3266.955680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3266.978718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.981534] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3267.076162] Lustre: lustre-OST0000: new disk, initializing [ 3267.077927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3267.103607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3268.486138] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3272.273324] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3274.740627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3274.743271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3279.703471] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3286.084714] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3292.052394] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3331.041016] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3331.042517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.043937] LustreError: Skipped 1 previous similar message [ 3331.043947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3331.043953] Lustre: Skipped 4 previous similar messages [ 3331.049438] Lustre: Skipped 1 previous similar message [ 3336.364156] Lustre: server umount lustre-OST0000 complete [ 3338.720062] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3338.722405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.521228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3351.524970] Lustre: Skipped 9 previous similar messages [ 3352.031149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3353.934774] LustreError: 235152:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759845975 with bad export cookie 3936041556640955202 [ 3353.941311] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3353.941745] LustreError: 235152:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3354.136773] Lustre: server umount lustre-MDT0001 complete [ 3354.138194] Lustre: Skipped 1 previous similar message [ 3357.605176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.264342] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3366.367252] Lustre: 240645:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759845982/real 1759845982] req@0000000023338b0a x1845332015118016/t0(0) o251->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759845988 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3371.045765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.212981] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.218716] LustreError: 241280:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3371.240450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.678964] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3375.990578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.495460] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3380.370175] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3382.512460] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3385.339914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3386.954960] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3389.835400] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3406.306622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.314558] Lustre: Skipped 1 previous similar message [ 3406.317793] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3406.318249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3406.325781] Lustre: Skipped 1 previous similar message [ 3408.954124] Lustre: server umount lustre-OST0000 complete [ 3408.956253] Lustre: Skipped 1 previous similar message [ 3416.544330] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3416.547980] LustreError: Skipped 1 previous similar message [ 3418.064508] LustreError: 244059:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846039 with bad export cookie 3936041556640957008 [ 3418.069058] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3418.073495] LustreError: 244059:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3423.662458] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3424.866234] Key type lgssc unregistered [ 3424.983395] LNet: 245378:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3424.988213] LNet: Removed LNI 192.168.202.109@tcp [ 3425.349576] Key type .llcrypt unregistered [ 3425.350647] Key type ._llcrypt unregistered [ 3441.574853] alg: No test for adler32 (adler32-zlib) [ 3442.326277] Key type ._llcrypt registered [ 3442.327166] Key type .llcrypt registered [ 3442.364161] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 3447.732027] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3448.280465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3448.438560] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3448.515458] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3448.517160] LNet: Accept secure, port 988 [ 3450.103159] Key type lgssc registered [ 3450.507134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3454.018227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3456.406725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3458.922267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3460.915895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3463.637649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3463.659382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.737176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3464.746356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3464.790607] Lustre: lustre-MDT0000: new disk, initializing [ 3464.813519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3464.819623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3466.328209] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3470.677775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3470.711203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.752182] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3470.780235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3470.783326] Lustre: Skipped 1 previous similar message [ 3470.824898] Lustre: lustre-MDT0001: new disk, initializing [ 3470.864287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3470.876888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3470.880417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3472.404772] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3476.782697] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3479.409175] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3482.133209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3482.154792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.157251] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3482.246460] Lustre: lustre-OST0000: new disk, initializing [ 3482.248231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3482.282356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3483.994842] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3488.493674] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3490.817081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3490.822762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3497.462644] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3501.905254] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3508.442838] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3557.346261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3557.347481] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3557.353514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3557.356491] LustreError: Skipped 1 previous similar message [ 3557.361494] Lustre: Skipped 1 previous similar message [ 3559.408433] Lustre: server umount lustre-OST0000 complete [ 3562.465571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.467036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3562.475646] Lustre: Skipped 2 previous similar messages [ 3566.726462] Lustre: server umount lustre-MDT0000 complete [ 3567.584790] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3567.593624] LustreError: Skipped 1 previous similar message [ 3568.487590] LustreError: 249448:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846190 with bad export cookie 7955897633566466072 [ 3568.493627] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3568.495355] LustreError: 249448:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3568.633339] Lustre: server umount lustre-MDT0001 complete [ 3572.334250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.957719] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3581.407254] Lustre: 254847:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759846197/real 1759846197] req@00000000e60c46fa x1845332352742528/t0(0) o251->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1759846203 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3581.428861] Lustre: server umount MGS complete [ 3586.534965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.698063] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.705856] LustreError: 255481:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3586.733787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3588.670890] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3592.161197] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3592.605737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.754578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3594.767910] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3598.232876] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3600.401183] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3603.194888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3603.337087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.202317] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3608.710704] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3638.240074] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3638.244587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3638.251984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3638.255423] Lustre: Skipped 1 previous similar message [ 3642.338045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3642.340093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3642.347689] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3644.280936] Lustre: server umount lustre-OST0000 complete [ 3647.457355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3647.458487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3647.463851] Lustre: Skipped 1 previous similar message [ 3647.466962] Lustre: Skipped 1 previous similar message [ 3652.576703] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3652.586050] LustreError: Skipped 1 previous similar message [ 3654.057554] LustreError: 257420:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846275 with bad export cookie 7955897633566467983 [ 3654.061384] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3654.066719] LustreError: 257420:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3654.217066] Lustre: server umount lustre-MDT0001 complete [ 3654.219578] Lustre: Skipped 1 previous similar message [ 3660.055287] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3661.517973] Key type lgssc unregistered [ 3661.638357] LNet: 259605:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3661.640628] LNet: Removed LNI 192.168.202.109@tcp [ 3661.992669] Key type .llcrypt unregistered [ 3661.994349] Key type ._llcrypt unregistered [ 3674.532382] alg: No test for adler32 (adler32-zlib) [ 3675.284333] Key type ._llcrypt registered [ 3675.285259] Key type .llcrypt registered [ 3675.322105] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3675.623741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3675.841941] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3675.911620] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3675.913631] LNet: Accept secure, port 988 [ 3677.495191] Key type lgssc registered [ 3677.875850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3680.853138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.018262] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3682.039252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3684.218336] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3687.397436] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.547307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3688.745889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3690.806224] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3693.703317] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3696.060289] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3699.245634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3699.442869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3701.380402] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3704.879780] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3708.182389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3708.224444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3708.229585] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3708.313652] Lustre: lustre-OST0001: new disk, initializing [ 3708.316578] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3708.374739] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3708.992631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3708.995549] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3709.012417] LustreError: 264150:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 3710.055720] LustreError: 264160:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3710.063625] LustreError: 264160:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 3710.248450] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3711.076391] LustreError: 264160:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3711.083663] LustreError: 264160:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 3713.132897] LustreError: 264150:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 3713.138443] LustreError: 264150:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 3716.256586] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3716.380225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3717.221415] LustreError: 264160:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3717.227574] LustreError: 264160:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 3718.991761] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9064c6ced800.ost_server_uuid 40 [ 3719.652886] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9064c6ced800.ost_server_uuid in FULL state after 0 sec [ 3722.220947] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9064c6ced800.ost_server_uuid 40 [ 3722.820118] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9064c6ced800.ost_server_uuid in FULL state after 0 sec [ 3739.110446] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3739.113669] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3739.114516] LustreError: Skipped 1 previous similar message [ 3739.122913] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3739.127318] Lustre: Skipped 1 previous similar message [ 3739.783373] Lustre: lustre-OST0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 3744.225890] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3744.227778] Lustre: Skipped 1 previous similar message [ 3749.343264] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3749.350538] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3749.354405] Lustre: Skipped 2 previous similar messages [ 3749.370815] Lustre: server umount lustre-OST0001 complete [ 3750.014665] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3754.466812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.467738] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3754.467976] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3754.471286] Lustre: Skipped 1 previous similar message [ 3754.482131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3754.485512] Lustre: Skipped 1 previous similar message [ 3757.758742] Lustre: server umount lustre-OST0000 complete [ 3759.585303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.591948] Lustre: Skipped 2 previous similar messages [ 3764.704662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3764.706514] Lustre: Skipped 3 previous similar messages [ 3765.740249] Lustre: server umount lustre-MDT0000 complete [ 3767.582795] LustreError: 261150:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846389 with bad export cookie 9061672150496591470 [ 3767.588846] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3767.590876] LustreError: 261150:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3767.789129] Lustre: server umount lustre-MDT0001 complete [ 3773.574706] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3775.178118] Key type lgssc unregistered [ 3775.340694] LNet: 266819:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3775.346331] LNet: Removed LNI 192.168.202.109@tcp [ 3775.746380] Key type .llcrypt unregistered [ 3775.747435] Key type ._llcrypt unregistered [ 3791.352722] alg: No test for adler32 (adler32-zlib) [ 3792.104318] Key type ._llcrypt registered [ 3792.105198] Key type .llcrypt registered [ 3792.141272] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3792.505412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3792.673865] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3792.746533] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3792.747839] LNet: Accept secure, port 988 [ 3794.327117] Key type lgssc registered [ 3794.641557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3798.199073] loop0: detected capacity change from 0 to 1099511627776 [ 3798.413137] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 3798.416209] *** VALIDATE xfs *** [ 3798.422516] XFS (loop0): Mounting V5 Filesystem [ 3798.436676] XFS (loop0): Ending clean mount [ 3799.077159] loop1: detected capacity change from 0 to 18691697672192 [ 3815.997129] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3824.297317] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 3828.198246] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3832.867549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3836.027560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3838.876208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3841.635735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3844.480869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3844.516145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3845.630995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3845.643113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3845.703600] Lustre: lustre-MDT0000: new disk, initializing [ 3845.731811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.740873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3847.385640] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3852.976128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3853.011664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.068342] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3853.096369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3853.099558] Lustre: Skipped 1 previous similar message [ 3853.146882] Lustre: lustre-MDT0001: new disk, initializing [ 3853.186336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3853.204483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3853.208916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3855.273841] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3860.085939] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3862.612104] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3865.277534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3865.320260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3865.325378] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3865.457202] Lustre: lustre-OST0000: new disk, initializing [ 3865.459266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3865.495985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3867.569735] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3870.734853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3870.738950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3872.486180] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3874.713973] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3874.851479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3876.875589] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3877.022523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3880.933958] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3880.935462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3880.938473] LustreError: Skipped 1 previous similar message [ 3880.946294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3880.951584] Lustre: Skipped 1 previous similar message [ 3886.052771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.056475] Lustre: Skipped 1 previous similar message [ 3891.172573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3891.176024] Lustre: Skipped 1 previous similar message [ 3892.191203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3892.293833] Lustre: server umount lustre-OST0000 complete [ 3894.240471] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3894.245336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.252462] Lustre: Skipped 1 previous similar message [ 3894.255688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3896.291450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.322719] Lustre: server umount lustre-MDT0000 complete [ 3901.409361] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3901.416882] LustreError: Skipped 1 previous similar message [ 3902.190550] LustreError: 272610:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846523 with bad export cookie 4116306508147064365 [ 3902.197368] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3902.197367] LustreError: 272610:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3902.364478] Lustre: server umount lustre-MDT0001 complete [ 3906.267147] XFS (loop0): Unmounting Filesystem [ 3908.425745] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 10:15:28 (1759846528) [ 3911.613181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.859592] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3911.901247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3914.071339] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3917.281341] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.308164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.457118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3920.412078] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3923.987155] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3926.689500] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3929.847053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3930.007356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3932.125062] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 3935.391349] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3952.614473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3952.615151] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3952.616474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3952.616512] Lustre: Skipped 2 previous similar messages [ 3952.621071] Lustre: Skipped 1 previous similar message [ 3956.183178] Lustre: server umount lustre-OST0000 complete [ 3959.776562] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3959.780392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3962.848799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3962.852532] Lustre: Skipped 3 previous similar messages [ 3964.165210] Lustre: server umount lustre-MDT0000 complete [ 3965.766890] LustreError: 277243:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759846587 with bad export cookie 4116306508147065219 [ 3965.774350] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3965.774352] LustreError: 277243:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3972.156892] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3973.735865] Key type lgssc unregistered [ 3973.899625] LNet: 280225:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3973.904826] LNet: Removed LNI 192.168.202.109@tcp [ 3974.423380] Key type .llcrypt unregistered [ 3974.424724] Key type ._llcrypt unregistered [ 3987.339688] alg: No test for adler32 (adler32-zlib) [ 3988.091322] Key type ._llcrypt registered [ 3988.092223] Key type .llcrypt registered [ 3988.127289] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3988.517044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3988.727890] Lustre: Lustre: Build Version: 2.15.7_3_g1fb2f06 [ 3988.808978] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3988.811892] LNet: Accept secure, port 988 [ 3990.391205] Key type lgssc registered [ 3990.780768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3993.911664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3995.132440] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3995.165488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3997.332867] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 4000.224312] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4001.486038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.639633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4003.610144] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 4006.917209] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4009.543964] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4012.805155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4012.979412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.238759] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 4019.074354] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4034.202197] hrtimer: interrupt took 3185744 ns [ 4040.704451] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.723401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4040.728275] Lustre: Skipped 1 previous similar message [ 4040.768797] Lustre: Skipped 1 previous similar message [ 4045.505896] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 4047.229560] Lustre: server umount lustre-MDT0001 complete [ 4050.566979] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.429679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4055.707320] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4055.734311] LustreError: Skipped 2 previous similar messages [ 4060.807823] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.824208] LustreError: Skipped 2 previous similar messages [ 4062.568284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4062.746769] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4062.754203] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4062.815308] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4063.173608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4067.774552] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 4068.328081] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4068.360367] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4068.383695] Lustre: Skipped 1 previous similar message [ 4071.929231] LustreError: 285593:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 4071.937631] LustreError: 285593:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4071.944704] LustreError: 285593:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4071.951411] LustreError: 281832:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 4072.010448] LustreError: 285596:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.109@tcp: cfg command failed: rc = -17 [ 4072.024770] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 4072.024770] [ 4373.147320] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 4373.152393] Lustre: Skipped 2 previous similar messages [ 4373.477086] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.490907] Lustre: Skipped 1 previous similar message [ 4378.238457] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 4378.241639] Lustre: Skipped 2 previous similar messages [ 4378.921851] Lustre: server umount lustre-MDT0001 complete [ 4382.027628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4383.364443] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.389495] LustreError: Skipped 2 previous similar messages [ 4388.690514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4388.766645] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4388.770302] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4388.818436] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4389.106038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4392.535903] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 4394.470362] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4394.475970] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4398.584671] LustreError: 294989:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 4398.596061] LustreError: 294989:0:(genops.c:522:class_register_device()) Skipped 1 previous similar message [ 4398.612123] LustreError: 294989:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4398.623061] LustreError: 294989:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4398.637939] LustreError: 281832:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 4398.640525] LustreError: 281832:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 4398.703948] LustreError: 294992:0:(obd_config.c:680:class_setup()) Device 8 already setup (type osp) [ 4398.715228] LustreError: 294992:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.109@tcp: cfg command failed: rc = -17 [ 4398.719436] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.109@tcp [ 4398.719436] [ 4700.851853] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 4700.855171] Lustre: Skipped 2 previous similar messages [ 4701.664078] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4701.668612] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.678055] Lustre: Skipped 1 previous similar message [ 4711.039906] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 4711.048095] Lustre: Skipped 5 previous similar messages [ 4711.397174] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4711.491189] Lustre: server umount lustre-MDT0001 complete [ 4711.908595] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4711.916200] LustreError: Skipped 6 previous similar messages [ 4713.449793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4719.166143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4719.259496] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4719.265929] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4719.310607] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4719.544302] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4723.031988] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 4724.714083] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4724.727577] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4724.730375] Lustre: Skipped 1 previous similar message [ 4726.269208] LustreError: 304623:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 4726.278642] LustreError: 304623:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4726.283431] LustreError: 304623:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4726.295108] LustreError: 281832:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 4726.297891] LustreError: 281832:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 4726.345290] LustreError: 304626:0:(lod_lov.c:228:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 4726.366633] LustreError: 304626:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.109@tcp: cfg command failed: rc = -114 [ 4726.377140] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 4726.377140] [ 5042.147644] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5042.149825] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5042.151033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5042.157464] Lustre: Skipped 2 previous similar messages [ 5046.822037] Lustre: server umount lustre-MDT0000 complete [ 5047.267842] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.278716] LustreError: Skipped 5 previous similar messages [ 5049.126757] LustreError: 281821:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759847670 with bad export cookie 10610065742733444001 [ 5049.129463] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5049.136815] LustreError: 281821:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5052.386377] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5055.428923] Lustre: server umount lustre-MDT0001 complete [ 5064.123318] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 5070.024549] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5075.610224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5079.693705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5083.076523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5086.527544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5090.932353] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 10:35:10 (1759847710) [ 5095.091903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5095.133567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5095.283960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5095.305691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5095.358144] Lustre: lustre-MDT0000: new disk, initializing [ 5095.418153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5095.439076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5097.761756] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 5104.669381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5104.716464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5104.765641] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5104.791808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5104.798782] Lustre: Skipped 1 previous similar message [ 5104.853523] Lustre: lustre-MDT0001: new disk, initializing [ 5104.927924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5104.954203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5104.957861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5107.538972] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 5114.655443] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5118.543715] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5123.033046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5123.081363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5123.086688] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5123.241357] Lustre: lustre-OST0000: new disk, initializing [ 5123.245575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5123.307868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5125.533676] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all 8 [ 5131.306622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5131.317665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5131.895856] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5139.947255] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5139.950967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.958715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5139.961232] Lustre: Skipped 4 previous similar messages [ 5140.055168] Lustre: server umount lustre-MDT0000 complete [ 5140.056538] Lustre: Skipped 1 previous similar message [ 5141.473432] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5141.477704] LustreError: Skipped 2 previous similar messages [ 5142.192768] LustreError: 316677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759847763 with bad export cookie 10610065742733445891 [ 5142.196567] LustreError: 166-1: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5142.199984] LustreError: 316677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5150.687162] Lustre: 319932:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759847766/real 1759847766] req@0000000036b6d70c x1845332919111488/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759847772 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5168.663694] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 5173.372916] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5179.321587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5183.879129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5187.503067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5191.110982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5192.013419] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5069 sec ========== 10:36:51 (1759847811) [ 5213.646724] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5214.996058] Key type lgssc unregistered [ 5215.155568] LNet: 325034:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5215.159713] LNet: Removed LNI 192.168.202.109@tcp [ 5215.567357] Key type .llcrypt unregistered [ 5215.569321] Key type ._llcrypt unregistered