[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474588690 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.032 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003164] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007554] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 0.009007] Calibrating delay loop (skipped) preset value.. 4800.06 BogoMIPS (lpj=2400032) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011006] LSM: Security Framework initializing [ 0.013043] Yama: becoming mindful. [ 0.014042] SELinux: Initializing. [ 0.015078] *** VALIDATE selinux *** [ 0.023799] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027713] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029065] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031091] *** VALIDATE tmpfs *** [ 0.032365] *** VALIDATE proc *** [ 0.033209] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039037] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.043885] debug: unmapping init [mem 0xffffffffa3459000-0xffffffffa3460fff] [ 0.045854] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046651] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054262] rcu: Hierarchical SRCU implementation. [ 0.056510] smp: Bringing up secondary CPUs ... [ 0.057574] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.064103] smp: Brought up 1 node, 4 CPUs [ 0.066016] smpboot: Max logical packages: 1 [ 0.067018] smpboot: Total of 4 processors activated (19200.25 BogoMIPS) [ 0.213224] node 0 deferred pages initialised in 144ms [ 0.216115] devtmpfs: initialized [ 0.217388] x86/mm: Memory block size: 128MB [ 0.220992] gcov: version magic: 0x41383552 [ 0.224292] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.228137] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.230334] pinctrl core: initialized pinctrl subsystem [ 0.233203] [ 0.233784] ************************************************************* [ 0.236013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.238012] ** ** [ 0.240015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.243011] ** ** [ 0.245012] ** This means that this kernel is built to expose internal ** [ 0.248013] ** IOMMU data structures, which may compromise security on ** [ 0.250012] ** your system. ** [ 0.253013] ** ** [ 0.254009] ** If you see this message and you are not debugging the ** [ 0.257013] ** kernel, report this immediately to your vendor! ** [ 0.259010] ** ** [ 0.261014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264013] ************************************************************* [ 0.266606] NET: Registered protocol family 16 [ 0.269319] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.272057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.274045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.277029] cpuidle: using governor menu [ 0.278529] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.281062] PCI: Using configuration type 1 for base access [ 0.283121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.290099] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.292035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.295159] cryptd: max_cpu_qlen set to 1000 [ 0.298098] ACPI: Added _OSI(Module Device) [ 0.299012] ACPI: Added _OSI(Processor Device) [ 0.300017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.302011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.306860] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.313366] ACPI: Interpreter enabled [ 0.314050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.316013] ACPI: Using IOAPIC for interrupt routing [ 0.317082] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.320300] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.329842] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.333043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.335020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.338071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.343341] acpiphp: Slot [2] registered [ 0.345130] acpiphp: Slot [5] registered [ 0.347101] acpiphp: Slot [6] registered [ 0.348135] acpiphp: Slot [7] registered [ 0.349105] acpiphp: Slot [8] registered [ 0.350099] acpiphp: Slot [9] registered [ 0.352109] acpiphp: Slot [10] registered [ 0.353060] acpiphp: Slot [3] registered [ 0.355111] acpiphp: Slot [4] registered [ 0.356108] acpiphp: Slot [11] registered [ 0.357085] acpiphp: Slot [12] registered [ 0.358097] acpiphp: Slot [13] registered [ 0.360145] acpiphp: Slot [14] registered [ 0.362081] acpiphp: Slot [15] registered [ 0.363083] acpiphp: Slot [16] registered [ 0.365107] acpiphp: Slot [17] registered [ 0.366095] acpiphp: Slot [18] registered [ 0.368113] acpiphp: Slot [19] registered [ 0.369142] acpiphp: Slot [20] registered [ 0.371136] acpiphp: Slot [21] registered [ 0.373122] acpiphp: Slot [22] registered [ 0.374081] acpiphp: Slot [23] registered [ 0.376128] acpiphp: Slot [24] registered [ 0.378121] acpiphp: Slot [25] registered [ 0.380107] acpiphp: Slot [26] registered [ 0.381099] acpiphp: Slot [27] registered [ 0.382068] acpiphp: Slot [28] registered [ 0.384082] acpiphp: Slot [29] registered [ 0.385085] acpiphp: Slot [30] registered [ 0.386101] acpiphp: Slot [31] registered [ 0.388080] PCI host bridge to bus 0000:00 [ 0.390012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.392027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.394027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.397030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.399021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.402023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.403154] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.407981] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.411474] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.420765] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.426019] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.431030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.433027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.436022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.438489] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.441788] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.444037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.447886] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.454013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.465012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.470016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.477994] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.485212] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.492025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.511023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.523306] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.532016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.541016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.571017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.582640] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.590023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.599021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.621021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.632133] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.637014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.641018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.654019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.662607] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.668015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.673015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.686021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.697226] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.711025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.719022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.742022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.754244] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.757682] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.761543] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.764381] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.767226] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.773173] iommu: Default domain type: Passthrough [ 0.774401] SCSI subsystem initialized [ 0.775000] ACPI: bus type USB registered [ 0.777103] usbcore: registered new interface driver usbfs [ 0.779056] usbcore: registered new interface driver hub [ 0.781095] usbcore: registered new device driver usb [ 0.784138] pps_core: LinuxPPS API ver. 1 registered [ 0.786010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.789056] PTP clock support registered [ 0.790228] EDAC MC: Ver: 3.0.0 [ 0.791438] PCI: Using ACPI for IRQ routing [ 0.793493] NetLabel: Initializing [ 0.794012] NetLabel: domain hash size = 128 [ 0.795011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.796197] NetLabel: unlabeled traffic allowed by default [ 0.799118] vgaarb: loaded [ 0.801296] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.803021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.811270] clocksource: Switched to clocksource kvm-clock [ 0.924314] VFS: Disk quotas dquot_6.6.0 [ 0.926151] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.929339] *** VALIDATE ramfs *** [ 0.930295] *** VALIDATE hugetlbfs *** [ 0.932204] pnp: PnP ACPI init [ 0.934685] pnp: PnP ACPI: found 6 devices [ 0.951536] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.955987] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.958666] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.960931] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.963916] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.966074] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.968726] NET: Registered protocol family 2 [ 0.971182] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.977781] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.982953] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.991342] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.995134] TCP: Hash tables configured (established 65536 bind 65536) [ 0.998543] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.001439] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.003609] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.006902] NET: Registered protocol family 1 [ 1.010345] RPC: Registered named UNIX socket transport module. [ 1.012467] RPC: Registered udp transport module. [ 1.014013] RPC: Registered tcp transport module. [ 1.016039] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.018984] NET: Registered protocol family 44 [ 1.021186] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.023793] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.026384] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.029424] PCI: CLS 0 bytes, default 64 [ 1.031196] Unpacking initramfs... [ 2.420221] debug: unmapping init [mem 0xffff9f42fcc54000-0xffff9f42fffbffff] [ 2.423976] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.426138] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.429602] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 2.950482] Initialise system trusted keyrings [ 2.952868] Key type blacklist registered [ 2.955261] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.966855] zbud: loaded [ 2.970350] *** VALIDATE nfs *** [ 2.971556] *** VALIDATE nfs4 *** [ 2.973038] pstore: using deflate compression [ 2.977205] Platform Keyring initialized [ 3.116538] NET: Registered protocol family 38 [ 3.118603] Key type asymmetric registered [ 3.120506] Asymmetric key parser 'x509' registered [ 3.122609] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.126124] io scheduler mq-deadline registered [ 3.128422] io scheduler kyber registered [ 3.130247] io scheduler bfq registered [ 3.133788] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.137671] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.140747] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.144398] ACPI: Power Button [PWRF] [ 3.151793] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.158953] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.217831] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.227226] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.246227] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.274327] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.302452] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.307691] Non-volatile memory driver v1.3 [ 3.309933] Linux agpgart interface v0.103 [ 3.344282] virtio_blk virtio1: [vda] 68000 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.349041] vda: detected capacity change from 0 to 34816000 [ 3.365816] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.369269] vdb: detected capacity change from 0 to 1073741824 [ 3.389470] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.392660] vdc: detected capacity change from 0 to 2621440000 [ 3.412729] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.416114] vdd: detected capacity change from 0 to 2621440000 [ 3.431500] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.435012] vde: detected capacity change from 0 to 4294967296 [ 3.453441] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.456829] vdf: detected capacity change from 0 to 4294967296 [ 3.466601] libphy: Fixed MDIO Bus: probed [ 3.475628] usbcore: registered new interface driver usbserial_generic [ 3.478347] usbserial: USB Serial support registered for generic [ 3.481663] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.486491] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.488446] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.491411] mousedev: PS/2 mouse device common for all mice [ 3.494842] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.497266] rtc_cmos 00:05: RTC can wake from S4 [ 3.503905] rtc_cmos 00:05: registered as rtc0 [ 3.506409] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.506825] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.509598] intel_pstate: CPU model not supported [ 3.516177] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.519267] hid: raw HID events driver (C) Jiri Kosina [ 3.523502] usbcore: registered new interface driver usbhid [ 3.525786] usbhid: USB HID core driver [ 3.527527] drop_monitor: Initializing network drop monitor service [ 3.529843] Initializing XFRM netlink socket [ 3.532057] NET: Registered protocol family 10 [ 3.534848] Segment Routing with IPv6 [ 3.536567] NET: Registered protocol family 17 [ 3.538867] mpls_gso: MPLS GSO support [ 3.545495] RAS: Correctable Errors collector initialized. [ 3.547575] AVX version of gcm_enc/dec engaged. [ 3.549544] AES CTR mode by8 optimization enabled [ 3.621510] sched_clock: Marking stable (3621479060, 0)->(4539909993, -918430933) [ 3.624269] registered taskstats version 1 [ 3.626392] Loading compiled-in X.509 certificates [ 3.628841] zswap: loaded using pool lzo/zbud [ 3.655395] Key type big_key registered [ 3.669419] Key type encrypted registered [ 3.671210] ima: No TPM chip found, activating TPM-bypass! [ 3.673166] ima: Allocated hash algorithm: sha1 [ 3.674934] ima: No architecture policies found [ 3.676770] evm: Initialising EVM extended attributes: [ 3.678824] evm: security.selinux [ 3.680435] evm: security.ima [ 3.681324] evm: security.capability [ 3.682613] evm: HMAC attrs: 0x1 [ 3.685949] rtc_cmos 00:05: setting system clock to 2026-04-14 19:42:17 UTC (1776195737) [ 3.692207] debug: unmapping init [mem 0xffffffffa4403000-0xffffffffa45fffff] [ 3.695543] debug: unmapping init [mem 0xffffffffa3182000-0xffffffffa3458fff] [ 3.707107] Write protecting the kernel read-only data: 28672k [ 3.710676] debug: unmapping init [mem 0xffffffffa1803000-0xffffffffa19fffff] [ 3.715525] debug: unmapping init [mem 0xffffffffa2114000-0xffffffffa21fffff] [ 3.752468] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.759037] systemd[1]: Detected virtualization kvm. [ 3.760778] systemd[1]: Detected architecture x86-64. [ 3.762196] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.787646] systemd[1]: No hostname configured. [ 3.789050] systemd[1]: Set hostname to . [ 3.790980] random: systemd: uninitialized urandom read (16 bytes read) [ 3.793087] systemd[1]: Initializing machine ID from random generator. [ 3.839705] random: ln: uninitialized urandom read (6 bytes read) [ 3.935562] random: systemd: uninitialized urandom read (16 bytes read) [ 3.938287] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.943190] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.947196] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.638790] device-mapper: uevent: version 1.0.3 [ 4.641837] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.337706] virtio_net virtio0 ens2: renamed from eth0 [ 5.365088] random: fast init done [ 5.382874] scsi host0: ata_piix [ 5.409872] scsi host1: ata_piix [ 5.411907] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.415342] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.713245] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.188946] random: crng init done [ 10.190615] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.554952] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.651516] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.903379] SELinux: Disabled at runtime. [ 11.966595] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.974946] systemd[1]: Detected virtualization kvm. [ 11.976777] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.438753] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.442592] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.449181] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.453820] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.457641] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.465748] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.475691] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ 12.593479] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.893083] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.183495] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.238413] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.310955] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.332423] EDAC sbridge: Ver: 1.1.2 [ 15.059410] Key type dns_resolver registered [ 15.359719] NFS: Registering the id_resolver key type [ 15.362476] Key type id_resolver registered [ 15.364445] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg345-server login: [ 36.794195] hrtimer: interrupt took 8374420 ns [ 46.720519] libcfs: loading out-of-tree module taints kernel. [ 46.794828] alg: No test for adler32 (adler32-zlib) [ 47.570655] Key type ._llcrypt registered [ 47.578477] Key type .llcrypt registered [ 47.681154] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 64.686875] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 65.905070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 66.445830] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 67.086994] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 67.091662] LNet: Accept secure, port 988 [ 68.826762] Key type lgssc registered [ 70.574903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.791199] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 92.107237] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.815055] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 109.202103] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.974007] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 131.100600] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 141.837409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.947754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.143897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 143.192869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 143.272497] Lustre: lustre-MDT0000: new disk, initializing [ 143.366265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.391540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 147.259321] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 159.051723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 159.133578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.187419] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 159.234461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 159.239225] Lustre: Skipped 1 previous similar message [ 159.345728] Lustre: lustre-MDT0001: new disk, initializing [ 159.421439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.454780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.465631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 163.707833] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 176.643502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.750247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 176.760597] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 177.016517] Lustre: lustre-OST0000: new disk, initializing [ 177.024817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 177.082408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.367456] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 184.377247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 184.384142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.760361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.850711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.859339] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 196.933446] Lustre: lustre-OST0001: new disk, initializing [ 196.940206] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 196.983808] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.699212] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 203.313561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.321317] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.171654] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 218.923514] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 227.785708] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing check_logdir /tmp/testlogs/ [ 232.644753] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing yml_node [ 236.117452] Lustre: DEBUG MARKER: Client: 2.15.8.4 [ 238.241786] Lustre: DEBUG MARKER: MDS: 2.15.8.4 [ 240.549525] Lustre: DEBUG MARKER: OSS: 2.15.8.4 [ 241.881226] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 15:46:14 EDT 2026 [ 249.442962] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 251.272722] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 269.792094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.793963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.805075] Lustre: Skipped 3 previous similar messages [ 269.820480] Lustre: Skipped 3 previous similar messages [ 274.106874] Lustre: server umount lustre-MDT0000 complete [ 274.914361] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 274.934988] LustreError: Skipped 3 previous similar messages [ 277.599655] LustreError: 9226:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196011 with bad export cookie 17915275427768847546 [ 277.603362] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 277.615758] LustreError: 9226:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 280.032604] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.035588] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.046658] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 280.049613] LustreError: Skipped 1 previous similar message [ 280.073019] Lustre: Skipped 1 previous similar message [ 282.082036] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 282.084548] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 282.093390] LustreError: Skipped 1 previous similar message [ 282.106590] Lustre: Skipped 2 previous similar messages [ 284.076918] Lustre: server umount lustre-MDT0001 complete [ 288.235222] Lustre: server umount lustre-OST0000 complete [ 292.695914] Lustre: server umount lustre-OST0001 complete [ 299.283289] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 307.450865] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 316.141364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 322.517095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.007263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 334.142568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 340.248340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 340.324712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 340.499324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 340.538835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 340.626367] Lustre: lustre-MDT0000: new disk, initializing [ 340.712620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 340.730436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 344.928684] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 355.023512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 355.107938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 355.239442] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 355.297593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 355.303231] Lustre: Skipped 1 previous similar message [ 355.429465] Lustre: lustre-MDT0001: new disk, initializing [ 355.522852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 355.577258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 355.591996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 359.815438] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 368.405575] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 374.352498] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 381.179213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 381.251762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 381.262819] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 381.449575] Lustre: lustre-OST0000: new disk, initializing [ 381.452943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 381.537291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.522716] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 395.045826] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 399.365382] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 399.577907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 404.806694] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 405.016809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 408.555811] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 408.557309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.579573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 408.594914] Lustre: Skipped 1 previous similar message [ 413.674763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 413.681194] Lustre: Skipped 1 previous similar message [ 421.343509] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 421.546542] Lustre: server umount lustre-OST0000 complete [ 427.493389] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 427.506237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 427.523788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 427.532204] Lustre: Skipped 3 previous similar messages [ 439.264201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 441.382919] Lustre: server umount lustre-MDT0000 complete [ 444.391207] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.099750] LustreError: 14338:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196178 with bad export cookie 17915275427768848995 [ 445.108542] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 445.114564] LustreError: 14338:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 450.120179] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 15:49:42 (1776196182) [ 456.533464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.966842] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.980161] LustreError: Skipped 1 previous similar message [ 457.026320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.229926] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 467.427520] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.442511] LustreError: Skipped 2 previous similar messages [ 469.876579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.085610] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 482.161873] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 487.536247] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 493.771143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 498.420803] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 500.350674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 500.359573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 505.447506] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 514.401530] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 521.698346] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 521.708672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.730658] Lustre: Skipped 1 previous similar message [ 521.738673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 521.743186] Lustre: Skipped 6 previous similar messages [ 526.780850] Lustre: server umount lustre-MDT0000 complete [ 526.787568] Lustre: Skipped 1 previous similar message [ 528.357690] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.379277] LustreError: Skipped 2 previous similar messages [ 530.789413] LustreError: 18732:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196264 with bad export cookie 17915275427768849856 [ 530.796737] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 530.803188] LustreError: 18732:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 541.663198] Lustre: 21860:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196269/real 1776196269] req@0000000095512ed9 x1862476287265792/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776196275 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 561.485773] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 571.142521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.464386] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.513212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.515766] Lustre: Skipped 2 previous similar messages [ 577.031715] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 585.649326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.597547] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 599.093869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 604.311037] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 614.593413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 614.665596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 614.679219] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 614.851850] Lustre: lustre-OST0001: new disk, initializing [ 614.869136] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 616.145297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 616.164072] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 620.761875] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 632.937363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 636.750678] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 652.257351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 652.271381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 652.274164] Lustre: Skipped 4 previous similar messages [ 652.291284] Lustre: Skipped 6 previous similar messages [ 657.637205] Lustre: server umount lustre-MDT0000 complete [ 657.641803] Lustre: Skipped 2 previous similar messages [ 660.947385] LustreError: 24035:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196394 with bad export cookie 17915275427768850703 [ 660.947803] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.955728] LustreError: 24035:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 669.535152] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196396/real 1776196396] req@0000000063809a91 x1862476287298240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776196403 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 669.566898] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.584235] Lustre: Skipped 1 previous similar message [ 670.687622] Lustre: 27062:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196398/real 1776196398] req@00000000441a782f x1862476287298368/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776196404 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 672.738456] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196399/real 1776196399] req@00000000bc5f5f0d x1862476287298432/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776196406 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 689.242044] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 698.604887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.964739] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 698.975521] LustreError: Skipped 2 previous similar messages [ 702.779638] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 711.109531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.619227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 711.627978] Lustre: Skipped 4 previous similar messages [ 716.417734] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 725.545491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 730.686485] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 739.026922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 743.965252] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 751.261194] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 754.860686] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 754.864385] Lustre: Skipped 1 previous similar message [ 767.970191] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 767.979361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.994096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 768.000564] Lustre: Skipped 4 previous similar messages [ 771.531574] Lustre: server umount lustre-MDT0000 complete [ 771.535396] Lustre: Skipped 3 previous similar messages [ 775.681509] LustreError: 28477:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196509 with bad export cookie 17915275427768851942 [ 775.703123] LustreError: 28477:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 775.706245] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 786.911555] Lustre: 32096:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196514/real 1776196514] req@000000004349f83b x1862476287324800/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776196520 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 786.952739] Lustre: 32096:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 799.084740] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 15:55:31 (1776196531) [ 822.765872] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 831.090433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 831.405288] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 831.415857] LustreError: Skipped 12 previous similar messages [ 835.253778] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 842.457380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.542911] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 854.832788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 860.004948] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 868.486646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 873.442661] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 881.305674] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 885.117435] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 899.052056] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 899.062754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.073830] Lustre: Skipped 4 previous similar messages [ 899.078787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 899.084820] Lustre: Skipped 4 previous similar messages [ 901.997773] Lustre: server umount lustre-MDT0000 complete [ 902.004144] Lustre: Skipped 3 previous similar messages [ 905.971630] LustreError: 36680:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196639 with bad export cookie 17915275427768853020 [ 905.982571] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 905.986148] LustreError: 36680:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 916.447189] Lustre: 37890:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196644/real 1776196644] req@0000000013411c68 x1862476287350912/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776196650 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 916.475250] Lustre: 37890:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 930.197766] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 15:57:42 (1776196662) [ 944.410624] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 953.134301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.993123] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 965.414517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.290491] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 978.760266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 979.126893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 979.132747] Lustre: Skipped 8 previous similar messages [ 983.738835] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 992.369483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 997.637175] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1005.264913] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1014.651801] Lustre: Setting parameter general.mdd.*.changelog_mask in log params [ 1014.660392] Lustre: Skipped 1 previous similar message [ 1026.833285] LustreError: 42576:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196760 with bad export cookie 17915275427768854091 [ 1026.838074] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1026.843922] LustreError: 42576:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1028.577916] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.592157] Lustre: Skipped 9 previous similar messages [ 1041.375130] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1048.441604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.241797] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xf89fd8151572664b to 0xf89fd81515726ad5 [ 1059.256696] Lustre: MGC192.168.203.145@tcp: Connection restored to 192.168.203.145@tcp (at 0@lo) [ 1059.679139] BUG: spinlock bad magic on CPU#3, lctl/43435 [ 1059.683689] lock: 0xffff9f437f8eed00, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 1059.688607] CPU: 3 PID: 43435 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1059.694629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 1059.708572] Call Trace: [ 1059.712151] ? dump_stack+0xbb/0x10e [ 1059.714875] ? spin_dump.cold.3+0x32/0x65 [ 1059.717758] ? do_raw_spin_lock+0xce/0x190 [ 1059.720314] ? _raw_spin_lock+0x21/0x30 [ 1059.721735] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 1059.728699] ? __seccomp_filter+0x4b/0x680 [ 1059.733807] ? full_proxy_write+0x5e/0xa0 [ 1059.738158] ? __vfs_write+0x1c/0x60 [ 1059.740675] ? vfs_write+0xd8/0x2b0 [ 1059.746605] ? ksys_write+0x66/0x120 [ 1059.751133] ? __x64_sys_write+0x1e/0x30 [ 1059.752693] ? do_syscall_64+0xc1/0x440 [ 1059.756038] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1063.358377] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1068.013496] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1070.354512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.061537] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1075.696400] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1075.703554] Lustre: Skipped 1 previous similar message [ 1080.935038] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1082.474725] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1087.893521] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1089.598461] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1096.165861] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1101.283858] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1101.296856] LustreError: Skipped 44 previous similar messages [ 1104.443503] LustreError: 39498:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196838 with bad export cookie 17915275427768855253 [ 1104.459881] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1113.503188] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196840/real 1776196840] req@00000000c89902cb x1862476287394688/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776196847 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 1113.530372] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1126.636669] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 16:00:58 (1776196858) [ 1142.453222] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1153.084613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.738668] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1170.934400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.560559] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1186.909393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1192.142980] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1201.256719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1205.796857] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1213.149893] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1237.988338] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1237.995910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1238.000765] Lustre: Skipped 24 previous similar messages [ 1241.768725] Lustre: server umount lustre-MDT0000 complete [ 1241.772817] Lustre: Skipped 9 previous similar messages [ 1245.370638] LustreError: 50298:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196979 with bad export cookie 17915275427768856030 [ 1245.377884] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.393866] LustreError: 50298:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1263.682907] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 16:03:15 (1776196995) [ 1266.117166] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1267.877548] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 16:03:19 (1776196999) [ 1278.411975] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 1286.612337] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1296.714291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1303.129574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1310.480561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1310.555416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1310.860866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1310.863158] Lustre: Skipped 2 previous similar messages [ 1310.918387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1311.036335] Lustre: lustre-MDT0000: new disk, initializing [ 1311.157245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1316.064894] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1326.773454] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1333.285364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1333.395969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1333.406669] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1333.714269] Lustre: lustre-OST0000: new disk, initializing [ 1333.719763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1333.732195] Lustre: Skipped 1 previous similar message [ 1338.653159] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1348.451344] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1350.511300] Lustre: DEBUG MARKER: create test files [ 1381.354261] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1381.355797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.382062] Lustre: Skipped 11 previous similar messages [ 1394.159512] Lustre: 55602:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197121/real 1776197121] req@0000000039ebe665 x1862476287434304/t0(0) o251->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776197127 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1394.189875] Lustre: 55602:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1408.405624] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1412.635732] Key type lgssc unregistered [ 1413.232350] LNet: 56206:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.252720] LNet: Removed LNI 192.168.203.145@tcp [ 1414.728263] Key type .llcrypt unregistered [ 1414.730953] Key type ._llcrypt unregistered [ 1442.215685] alg: No test for adler32 (adler32-zlib) [ 1442.973735] Key type ._llcrypt registered [ 1442.976158] Key type .llcrypt registered [ 1443.096099] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1444.200840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1444.478554] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 1444.666272] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1444.670506] LNet: Accept secure, port 988 [ 1446.344594] Key type lgssc registered [ 1447.379672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1455.147572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.714723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.377196] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1470.132653] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1477.048054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1477.328952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1481.208900] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1486.576229] Lustre: lustre-OST0000: deleting orphan objects from 0x0:102 to 0x0:129 [ 1486.669972] Lustre: 59050:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1486.679375] Lustre: 59050:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1486.688937] Lustre: 59050:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1486.695031] Lustre: 59050:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1486.706950] Lustre: 59050:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1486.718512] Lustre: 59050:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1489.418097] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1493.989248] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1498.840396] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1507.300393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.301783] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1507.318530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1512.418506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1517.537208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1520.095636] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1520.284155] Lustre: server umount lustre-OST0000 complete [ 1531.361105] Lustre: 60070:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197259/real 1776197259] req@000000008af4f962 x1862477732125504/t0(0) o251->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776197265 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1531.643673] Lustre: server umount lustre-MDT0000 complete [ 1545.629718] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1548.124862] Key type lgssc unregistered [ 1548.420717] LNet: 60675:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.430284] LNet: Removed LNI 192.168.203.145@tcp [ 1549.189696] Key type .llcrypt unregistered [ 1549.193219] Key type ._llcrypt unregistered [ 1577.696538] alg: No test for adler32 (adler32-zlib) [ 1578.449406] Key type ._llcrypt registered [ 1578.451955] Key type .llcrypt registered [ 1578.575285] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1579.453710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1580.075611] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 1580.259692] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1580.263691] LNet: Accept secure, port 988 [ 1581.936187] Key type lgssc registered [ 1582.864712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1590.251726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.866152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.942761] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1603.339416] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1609.725253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1610.035130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.370778] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1621.290973] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 1621.347022] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1621.362175] Lustre: 63620:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1621.379598] Lustre: 63620:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1621.387246] Lustre: 63620:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1621.400577] Lustre: 63620:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1621.408603] Lustre: 63620:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1621.416357] Lustre: 63620:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1626.787637] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1636.834154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.849130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1641.953416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1642.121196] Lustre: server umount lustre-OST0000 complete [ 1652.191833] Lustre: 64583:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197379/real 1776197379] req@000000002c30599e x1862477874731392/t0(0) o251->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776197385 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1652.391667] Lustre: server umount lustre-MDT0000 complete [ 1670.028612] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1672.904630] Key type lgssc unregistered [ 1673.211620] LNet: 65206:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1673.223573] LNet: Removed LNI 192.168.203.145@tcp [ 1674.195546] Key type .llcrypt unregistered [ 1674.198758] Key type ._llcrypt unregistered [ 1697.817935] alg: No test for adler32 (adler32-zlib) [ 1698.570391] Key type ._llcrypt registered [ 1698.572277] Key type .llcrypt registered [ 1698.687923] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 1714.447213] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1715.952639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1716.920477] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 1717.216164] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1717.235078] LNet: Accept secure, port 988 [ 1718.927138] Key type lgssc registered [ 1720.164622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1730.688097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1737.216211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1744.286982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1750.949902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1757.653720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1757.715151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1759.048976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1759.107470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1759.190387] Lustre: lustre-MDT0000: new disk, initializing [ 1759.320837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.387312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1764.722732] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1774.917617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1774.987182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1775.076937] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1775.129855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1775.136788] Lustre: Skipped 1 previous similar message [ 1775.219072] Lustre: lustre-MDT0001: new disk, initializing [ 1775.292978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1775.322295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1775.332209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1779.337047] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1789.228243] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1795.641792] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1802.260217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1802.319305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1802.327577] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1802.578317] Lustre: lustre-OST0000: new disk, initializing [ 1802.584960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1802.664721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.509173] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1808.961250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1808.973176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1817.102116] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1821.670388] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1821.855512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1826.507886] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1826.685529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1829.856651] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1829.857278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.868575] LustreError: Skipped 1 previous similar message [ 1829.869175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1829.891687] Lustre: Skipped 1 previous similar message [ 1829.907622] Lustre: Skipped 1 previous similar message [ 1834.790987] Lustre: server umount lustre-OST0000 complete [ 1840.096706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.099962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1840.130837] Lustre: Skipped 1 previous similar message [ 1840.163537] Lustre: Skipped 1 previous similar message [ 1844.742787] Lustre: server umount lustre-MDT0000 complete [ 1845.216420] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1845.233742] LustreError: Skipped 1 previous similar message [ 1846.241160] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1846.263433] LustreError: Skipped 1 previous similar message [ 1848.481877] LustreError: 68512:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197582 with bad export cookie 11895140200405742623 [ 1848.485155] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1848.491140] LustreError: 68512:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1848.804780] Lustre: server umount lustre-MDT0001 complete [ 1857.555111] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 16:13:09 (1776197589) [ 1868.036834] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1881.072313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1906.479750] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 1914.526376] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1926.137699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1934.207808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1941.697472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1948.759403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1956.860486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1956.992292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.252494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1957.302453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1957.357431] Lustre: lustre-MDT0000: new disk, initializing [ 1957.469424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.491049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1962.419136] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1974.409039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1974.483315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.597149] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1974.649438] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1974.653207] Lustre: Skipped 1 previous similar message [ 1974.744948] Lustre: lustre-MDT0001: new disk, initializing [ 1974.872824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1974.918564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1974.936735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1979.705857] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 1990.383736] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1997.502705] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2004.450108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2004.549163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2004.558290] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2004.883461] Lustre: lustre-OST0000: new disk, initializing [ 2004.893979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2004.987102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.802418] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2013.279863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2013.301311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2019.036172] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2024.390889] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2024.671395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2030.697499] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2030.988639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2034.146903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.156203] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2034.158252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2034.178490] Lustre: Skipped 1 previous similar message [ 2039.272077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2039.284060] Lustre: Skipped 1 previous similar message [ 2039.466160] Lustre: server umount lustre-OST0000 complete [ 2044.387769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.390285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2044.410668] Lustre: Skipped 2 previous similar messages [ 2044.435930] Lustre: Skipped 1 previous similar message [ 2054.624788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2054.630474] Lustre: Skipped 3 previous similar messages [ 2058.207464] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2058.466567] Lustre: server umount lustre-MDT0000 complete [ 2059.747018] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.766242] LustreError: Skipped 1 previous similar message [ 2061.967240] LustreError: 76809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197795 with bad export cookie 11895140200405743477 [ 2061.971509] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2061.983160] LustreError: 76809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2070.921808] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 16:16:42 (1776197802) [ 2077.130864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.416149] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2077.429533] LustreError: Skipped 2 previous similar messages [ 2077.502960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.677185] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2082.787892] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.465562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.596526] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2103.436953] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2109.020923] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2115.232338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2115.561491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.570153] Lustre: Skipped 1 previous similar message [ 2119.606660] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2126.605466] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2130.589310] Lustre: 84118:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting 4060a1d4-500b-41b5-a0f2-f980a1d1ae15 at adminstrative request [ 2132.968182] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.989396] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2139.111375] Lustre: MGS: Client 4060a1d4-500b-41b5-a0f2-f980a1d1ae15 (at 0@lo) reconnecting [ 2139.123146] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xa51406fd36a73acb to 0xa51406fd36a73d6b [ 2139.140590] Lustre: MGC192.168.203.145@tcp: Connection restored to 192.168.203.145@tcp (at 0@lo) [ 2167.359071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2167.453052] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2167.463274] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2167.560319] Lustre: lustre-OST0001: new disk, initializing [ 2167.565393] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2172.185871] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2176.583533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2176.593874] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2180.720867] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2187.243056] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.249251] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2187.267399] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2187.267747] Lustre: Skipped 1 previous similar message [ 2187.294924] LustreError: Skipped 1 previous similar message [ 2189.263286] Lustre: server umount lustre-OST0001 complete [ 2189.272130] Lustre: Skipped 1 previous similar message [ 2192.357990] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2192.360577] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.384894] LustreError: Skipped 2 previous similar messages [ 2202.593495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.610485] Lustre: Skipped 3 previous similar messages [ 2207.941955] Lustre: server umount lustre-MDT0000 complete [ 2207.944322] Lustre: Skipped 1 previous similar message [ 2210.966280] LustreError: 85729:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197944 with bad export cookie 11895140200405745003 [ 2210.973610] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2210.974768] LustreError: 85729:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2219.473925] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 16:19:11 (1776197951) [ 2221.035993] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2222.821486] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 16:19:14 (1776197954) [ 2224.334612] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2226.005455] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 16:19:18 (1776197958) [ 2227.453949] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2229.248547] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 16:19:21 (1776197961) [ 2230.828739] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2236.026690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2237.099126] loop: module loaded [ 2237.123824] loop0: detected capacity change from 0 to 4294967296 [ 2237.192878] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2238.883763] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2242.921502] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2246.821659] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2255.516488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2259.096474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2262.920685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2267.017513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2273.807421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2273.910852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2273.923421] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2273.938573] LustreError: 88566:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2273.948719] LustreError: 88566:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2273.954953] LustreError: 88566:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2273.962729] LustreError: 88566:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2291.302781] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2300.186454] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2309.544592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2316.220394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2322.692172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2329.305436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2336.441812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2336.534048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.051400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2337.107788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2337.204735] Lustre: lustre-MDT0000: new disk, initializing [ 2337.287226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.295658] Lustre: Skipped 1 previous similar message [ 2337.319415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2342.644138] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2354.267302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2354.392394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.530153] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2354.886272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2359.790673] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2370.284509] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2376.373040] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2382.959330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2383.009499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2383.016490] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2387.681618] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2391.619761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2397.563865] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2402.467433] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2402.717492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2407.807579] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2408.015817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2412.513293] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2412.519296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.524791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2412.531538] Lustre: Skipped 1 previous similar message [ 2412.542992] Lustre: Skipped 10 previous similar messages [ 2424.287172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2424.383722] Lustre: server umount lustre-OST0000 complete [ 2424.386668] Lustre: Skipped 1 previous similar message [ 2442.207773] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2443.239370] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.259381] LustreError: Skipped 3 previous similar messages [ 2446.568445] LustreError: 91421:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198180 with bad export cookie 11895140200405745584 [ 2446.573742] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2446.589651] LustreError: 91421:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2457.102630] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 16:23:08 (1776198188) [ 2464.335701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.196366] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2475.499463] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.512608] LustreError: Skipped 2 previous similar messages [ 2479.547760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.945585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2479.955426] Lustre: Skipped 3 previous similar messages [ 2485.059988] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2492.915038] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2499.504158] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2505.845940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2510.578082] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2517.790882] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2524.213692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2524.274273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.279391] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2524.376168] Lustre: lustre-OST0001: new disk, initializing [ 2524.379149] Lustre: Skipped 2 previous similar messages [ 2524.382929] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2524.392468] Lustre: Skipped 3 previous similar messages [ 2528.823251] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2529.848667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2529.864371] Lustre: Skipped 2 previous similar messages [ 2529.867631] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2537.472256] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2554.260859] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2575.651377] Lustre: Failing over lustre-MDT0000 [ 2575.777067] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 2575.783315] Lustre: Skipped 10 previous similar messages [ 2575.971666] Lustre: server umount lustre-MDT0000 complete [ 2575.973437] Lustre: Skipped 2 previous similar messages [ 2576.354421] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.360011] Lustre: Skipped 4 previous similar messages [ 2576.361714] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.374602] LustreError: Skipped 3 previous similar messages [ 2583.519248] Lustre: 66739:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198310/real 1776198310] req@00000000b1c918bf x1862478017457664/t0(0) o400->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776198317 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2583.565898] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2607.755819] LDISKFS-fs (dm-0): recovery complete [ 2607.764813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.305326] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xa51406fd36a74306 to 0xa51406fd36a7d0e9 [ 2616.339550] Lustre: MGC192.168.203.145@tcp: Connection restored to (at 0@lo) [ 2616.859327] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2620.803790] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2620.803862] Lustre: 99668:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2621.935073] Lustre: 97430:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2621.961396] Lustre: 97430:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 1 previous similar message [ 2621.967818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2622.074659] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2622.302325] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2622.401940] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2622.768422] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2622.778542] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2623.118534] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2623.133570] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2623.150908] Lustre: 101168:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2623.174825] Lustre: 101168:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 7 previous similar messages [ 2623.847959] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2623.865697] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 2624.205137] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2624.219485] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 2625.202425] Lustre: 101168:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2625.219878] Lustre: 101168:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 5 previous similar messages [ 2625.881094] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2625.894456] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 5 previous similar messages [ 2626.207218] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2626.213342] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 5 previous similar messages [ 2629.401578] Lustre: 101168:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2629.415647] Lustre: 101168:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages [ 2630.058678] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2630.087199] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2630.407403] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2630.417728] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2637.448196] Lustre: 101168:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2637.457816] Lustre: 101168:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 24 previous similar messages [ 2638.114779] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2638.135278] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 24 previous similar messages [ 2638.463275] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2638.469980] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 24 previous similar messages [ 2653.711671] Lustre: 101168:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2653.728718] Lustre: 101168:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 50 previous similar messages [ 2654.386548] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2654.405461] LustreError: 101168:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2654.727178] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2654.735375] LustreError: 101168:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2680.047145] Lustre: 101168:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2680.053198] Lustre: 101168:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2680.058327] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2680.067450] LustreError: 101168:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 2680.085096] Lustre: 101168:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@00000000f33b1fb5 x1862478000583232/t0(8589934778) o35->f233b14f-5320-4f73-9f2a-90a65be8046d@192.168.203.45@tcp:556/0 lens 392/0 e 0 to 0 dl 1776198456 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 2680.113613] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 2680.119564] Lustre: Skipped 2 previous similar messages [ 2680.121251] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2680.165778] Lustre: lustre-OST0000: deleting orphan objects from 0x0:79 to 0x0:577 [ 2680.170099] Lustre: lustre-OST0001: deleting orphan objects from 0x0:48 to 0x0:545 [ 2691.041061] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2691.050869] LustreError: Skipped 1 previous similar message [ 2691.054091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.067264] Lustre: Skipped 1 previous similar message [ 2703.846221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2703.851211] Lustre: Skipped 6 previous similar messages [ 2705.375391] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2705.421670] Lustre: server umount lustre-OST0000 complete [ 2708.962538] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.984858] LustreError: Skipped 61 previous similar messages [ 2727.890353] LustreError: 95967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198461 with bad export cookie 11895140200405782761 [ 2727.893505] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2727.900815] LustreError: 95967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2735.626438] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 16:27:47 (1776198467) [ 2742.029479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2742.036747] Lustre: *** cfs_fail_loc=197, val=0*** [ 2759.648398] LustreError: 103202:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 1: rc = -5 [ 2770.912883] LustreError: 103202:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 2776.098699] Lustre: *** cfs_fail_loc=197, val=0*** [ 2776.100385] Lustre: Skipped 65 previous similar messages [ 2799.840294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2799.852361] Lustre: Skipped 3 previous similar messages [ 2806.608035] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2814.209846] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2824.634133] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 16:29:16 (1776198556) [ 2845.216819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.850290] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2858.407181] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2862.349466] LustreError: 104859:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2862.362180] LustreError: 104859:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2862.375228] LustreError: 104859:0:(lod_dev.c:437:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 2868.703688] Lustre: 105550:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198596/real 1776198596] req@00000000bfe1bc06 x1862478017494080/t0(0) o251->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776198602 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2881.853184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2899.446855] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2908.547802] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2918.581377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2925.103103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2930.972952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2937.480827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2943.943307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2944.028457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.312059] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2944.380379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2944.519827] Lustre: lustre-MDT0000: new disk, initializing [ 2944.624616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2948.632586] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2958.640714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2958.699695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.078484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2963.579656] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 2972.589616] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2978.936309] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2985.772884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2985.871711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2985.877592] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2992.155472] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3002.011266] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3007.540255] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3007.718399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3012.656964] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3012.860903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3015.143738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3015.152219] Lustre: Skipped 5 previous similar messages [ 3015.157668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3015.162238] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3015.163713] Lustre: Skipped 8 previous similar messages [ 3015.167426] LustreError: Skipped 1 previous similar message [ 3020.831697] Lustre: server umount lustre-OST0000 complete [ 3020.835402] Lustre: Skipped 5 previous similar messages [ 3034.093803] LustreError: 109187:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198767 with bad export cookie 11895140200405793422 [ 3034.103248] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3034.109867] LustreError: 109187:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3042.437306] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 16:32:54 (1776198774) [ 3051.757796] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3054.079924] Key type lgssc unregistered [ 3054.338592] LNet: 113872:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3054.345297] LNet: Removed LNI 192.168.203.145@tcp [ 3055.145715] Key type .llcrypt unregistered [ 3055.148712] Key type ._llcrypt unregistered [ 3072.542453] alg: No test for adler32 (adler32-zlib) [ 3073.307289] Key type ._llcrypt registered [ 3073.316603] Key type .llcrypt registered [ 3073.438358] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 3085.334441] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3086.049270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3086.535621] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 3086.723485] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3086.727763] LNet: Accept secure, port 988 [ 3088.407771] Key type lgssc registered [ 3089.424698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3097.114620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3103.016929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3108.393431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3113.723726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3117.798866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3122.630392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3122.690641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3123.925874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3123.955072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3124.019330] Lustre: lustre-MDT0000: new disk, initializing [ 3124.094398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3124.112370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3128.556089] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3137.764596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3137.849623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3137.856779] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3138.052291] Lustre: lustre-OST0000: new disk, initializing [ 3138.057514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3138.065383] Lustre: Skipped 1 previous similar message [ 3138.107974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.791160] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3150.512138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3150.574969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3150.584608] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3150.675955] Lustre: lustre-OST0001: new disk, initializing [ 3150.680153] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3150.732330] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3154.360644] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3164.641793] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3164.648383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3164.658411] Lustre: Skipped 1 previous similar message [ 3167.365571] Lustre: server umount lustre-MDT0000 complete [ 3172.966410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3176.799210] Lustre: 115372:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198903/real 1776198903] req@000000001f7f41f1 x1862479453891520/t0(0) o400->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776198910 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3176.830910] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3177.433031] Lustre: server umount lustre-OST0000 complete [ 3180.772760] Lustre: server umount lustre-OST0001 complete [ 3188.441349] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 16:35:20 (1776198920) [ 3196.907862] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3209.439059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3210.633695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3217.471547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3218.367342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3225.576651] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 16:35:57 (1776198957) [ 3238.571120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3240.672584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3242.893891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3244.975966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3245.985275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3247.805127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3262.917929] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 3270.521980] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3280.325479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3288.942390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3295.890794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3302.910136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3309.663347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3309.743721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3309.949616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3309.991980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3310.077198] Lustre: lustre-MDT0000: new disk, initializing [ 3310.194301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.225441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3314.790943] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3325.433386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3325.501192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.551909] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3325.594033] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3325.600063] Lustre: Skipped 1 previous similar message [ 3325.671345] Lustre: lustre-MDT0001: new disk, initializing [ 3325.747882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3325.806061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3325.819581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3331.364532] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3342.896041] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3349.804487] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3357.455289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3357.546389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3357.552546] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3357.766808] Lustre: lustre-OST0000: new disk, initializing [ 3357.773704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3357.852562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3362.530372] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3365.446155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3365.454382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3371.730318] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3376.167079] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3376.431660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3381.219694] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3381.492589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3386.336480] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3386.339397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.344838] LustreError: Skipped 1 previous similar message [ 3386.362612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3386.380856] Lustre: Skipped 1 previous similar message [ 3386.409970] Lustre: Skipped 2 previous similar messages [ 3389.634587] Lustre: server umount lustre-OST0000 complete [ 3396.580884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.584752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3396.590768] Lustre: Skipped 1 previous similar message [ 3396.600178] Lustre: Skipped 1 previous similar message [ 3401.700135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3401.711648] Lustre: Skipped 1 previous similar message [ 3407.328047] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3407.433614] Lustre: server umount lustre-MDT0000 complete [ 3407.844967] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.860805] LustreError: Skipped 1 previous similar message [ 3410.493415] LustreError: 125904:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199144 with bad export cookie 8268111047493900740 [ 3410.500421] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3410.507804] LustreError: 125904:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3419.276795] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 16:39:11 (1776199151) [ 3425.765290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.271919] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.348958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.483634] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3431.397516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.519753] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.064365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.423220] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3449.680118] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3454.329972] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3460.557586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.769186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3460.773526] Lustre: Skipped 1 previous similar message [ 3464.751813] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3472.267438] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3481.886656] Lustre: *** cfs_fail_loc=159, val=0*** [ 3481.891149] Lustre: Skipped 5 previous similar messages [ 3489.198137] Lustre: lustre-MDT0000: Client e96f26da-934b-43fd-9fc7-b0801119495a (at 192.168.203.45@tcp) reconnecting [ 3490.360690] Lustre: *** cfs_fail_loc=159, val=0*** [ 3490.368606] Lustre: Skipped 1 previous similar message [ 3496.921671] Lustre: lustre-MDT0000: Client e96f26da-934b-43fd-9fc7-b0801119495a (at 192.168.203.45@tcp) reconnecting [ 3501.029383] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3501.045754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.060268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3501.070975] Lustre: Skipped 2 previous similar messages [ 3503.082949] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3506.190169] Lustre: server umount lustre-OST0000 complete [ 3506.196093] Lustre: Skipped 1 previous similar message [ 3511.265347] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3511.273835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3511.294752] Lustre: Skipped 1 previous similar message [ 3511.302662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3511.314582] Lustre: Skipped 2 previous similar messages [ 3518.433764] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.447040] LustreError: Skipped 1 previous similar message [ 3519.474487] LustreError: 132414:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199253 with bad export cookie 8268111047493901594 [ 3519.477938] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3519.493132] LustreError: 132414:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3530.706565] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3533.010761] Key type lgssc unregistered [ 3533.229708] LNet: 134314:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3533.244359] LNet: Removed LNI 192.168.203.145@tcp [ 3533.881012] Key type .llcrypt unregistered [ 3533.882602] Key type ._llcrypt unregistered [ 3551.047179] alg: No test for adler32 (adler32-zlib) [ 3551.807579] Key type ._llcrypt registered [ 3551.815164] Key type .llcrypt registered [ 3551.912390] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3552.818209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3553.085448] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 3553.237292] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3553.241190] LNet: Accept secure, port 988 [ 3554.887109] Key type lgssc registered [ 3555.671380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3562.256900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.719640] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3563.794156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3568.182519] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3569.125567] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3574.241742] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3576.085849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.444746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3580.588411] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3587.453572] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3591.897787] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3597.703364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3598.004282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3601.551100] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3609.000620] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3613.364898] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3622.875447] Lustre: *** cfs_fail_loc=159, val=0*** [ 3630.037464] Lustre: lustre-MDT0000: Client b07102b7-a68a-4f4e-98b7-4097779ec7bd (at 192.168.203.45@tcp) reconnecting [ 3632.226303] Lustre: *** cfs_fail_loc=159, val=0*** [ 3638.725699] Lustre: lustre-MDT0001: Client b07102b7-a68a-4f4e-98b7-4097779ec7bd (at 192.168.203.45@tcp) reconnecting [ 3639.717754] Lustre: *** cfs_fail_loc=159, val=0*** [ 3639.725598] Lustre: Skipped 3 previous similar messages [ 3646.901426] Lustre: lustre-MDT0001: Client b07102b7-a68a-4f4e-98b7-4097779ec7bd (at 192.168.203.45@tcp) reconnecting [ 3654.492824] Lustre: *** cfs_fail_loc=159, val=0*** [ 3654.500051] Lustre: Skipped 4 previous similar messages [ 3660.713338] Lustre: lustre-MDT0000: Client 5f06a082-435f-4909-9092-c563354c7873 (at 192.168.203.45@tcp) reconnecting [ 3661.806590] Lustre: *** cfs_fail_loc=159, val=0*** [ 3661.808572] Lustre: Skipped 13 previous similar messages [ 3668.938774] Lustre: lustre-MDT0000: Client 5f06a082-435f-4909-9092-c563354c7873 (at 192.168.203.45@tcp) reconnecting [ 3675.110845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.118496] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3675.135821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3675.151619] Lustre: Skipped 1 previous similar message [ 3676.472101] Lustre: server umount lustre-OST0000 complete [ 3680.225946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3680.241916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3680.250646] Lustre: Skipped 2 previous similar messages [ 3680.265765] Lustre: Skipped 1 previous similar message [ 3685.345672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3685.354124] Lustre: Skipped 1 previous similar message [ 3688.236497] Lustre: server umount lustre-MDT0000 complete [ 3690.464387] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3690.481133] LustreError: Skipped 1 previous similar message [ 3691.685351] LustreError: 135760:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199425 with bad export cookie 12699513446754110763 [ 3691.696989] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3691.705860] LustreError: 135760:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3691.946561] Lustre: server umount lustre-MDT0001 complete [ 3702.867974] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3705.489809] Key type lgssc unregistered [ 3705.746871] LNet: 140168:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3705.761073] LNet: Removed LNI 192.168.203.145@tcp [ 3706.524822] Key type .llcrypt unregistered [ 3706.530140] Key type ._llcrypt unregistered [ 3728.223350] alg: No test for adler32 (adler32-zlib) [ 3728.977561] Key type ._llcrypt registered [ 3728.980448] Key type .llcrypt registered [ 3729.078919] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3730.160864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3730.500040] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 3730.708672] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3730.716859] LNet: Accept secure, port 988 [ 3732.343104] Key type lgssc registered [ 3733.155147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3740.070079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3741.502854] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.568518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3745.406385] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3746.787262] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3751.918750] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.890806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3753.136374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3756.949171] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3764.295494] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3770.286971] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3776.544941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3776.896674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3780.934717] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3787.195660] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 3787.565322] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3791.292695] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:129 [ 3802.014772] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 3802.875909] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight in log params [ 3806.258354] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 3812.839562] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3812.851841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.870399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3816.936865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3816.942992] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3816.958235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.209106] Lustre: server umount lustre-OST0000 complete [ 3822.049319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.055138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3822.067263] Lustre: Skipped 1 previous similar message [ 3822.077505] Lustre: Skipped 2 previous similar messages [ 3827.181101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3827.192332] Lustre: Skipped 1 previous similar message [ 3832.289124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3832.295832] Lustre: Skipped 1 previous similar message [ 3835.871439] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3836.156029] Lustre: server umount lustre-MDT0000 complete [ 3837.409386] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.444482] LustreError: Skipped 1 previous similar message [ 3839.980097] LustreError: 141613:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199573 with bad export cookie 1397393229201887077 [ 3839.982456] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3839.997409] LustreError: 141613:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3840.292640] Lustre: server umount lustre-MDT0001 complete [ 3851.576671] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3854.086425] Key type lgssc unregistered [ 3854.354710] LNet: 145695:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3854.367333] LNet: Removed LNI 192.168.203.145@tcp [ 3855.157887] Key type .llcrypt unregistered [ 3855.160750] Key type ._llcrypt unregistered [ 3878.442678] alg: No test for adler32 (adler32-zlib) [ 3879.198091] Key type ._llcrypt registered [ 3879.205292] Key type .llcrypt registered [ 3879.373300] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3880.769934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3881.278400] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 3881.535873] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3881.545578] LNet: Accept secure, port 988 [ 3883.199131] Key type lgssc registered [ 3884.156717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3891.940655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.360353] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3893.425958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.364799] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3898.849920] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.972544] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.061740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3905.421442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3909.507217] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3917.731436] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3923.934967] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3932.044298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3932.549672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3935.594137] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 3937.842838] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 3943.803836] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:161 [ 3946.196771] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3952.309564] Lustre: *** cfs_fail_loc=159, val=0*** [ 3958.692479] Lustre: lustre-MDT0000: Client 3ee983f9-b291-41d0-acb5-4a80b8bf0f85 (at 192.168.203.45@tcp) reconnecting [ 3964.384757] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3964.397618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.410360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3964.425118] Lustre: Skipped 1 previous similar message [ 3964.440986] Lustre: Skipped 1 previous similar message [ 3965.785378] Lustre: server umount lustre-OST0000 complete [ 3969.504540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3969.511017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3969.514583] Lustre: Skipped 1 previous similar message [ 3969.526202] Lustre: Skipped 1 previous similar message [ 3974.624340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3974.631228] Lustre: Skipped 1 previous similar message [ 3975.439358] Lustre: server umount lustre-MDT0000 complete [ 3979.046339] LustreError: 147154:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199712 with bad export cookie 3487625115123926197 [ 3979.048419] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3979.063149] LustreError: 147154:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3979.454691] Lustre: server umount lustre-MDT0001 complete [ 3990.951133] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3993.727752] Key type lgssc unregistered [ 3994.010660] LNet: 150938:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3994.019746] LNet: Removed LNI 192.168.203.145@tcp [ 3994.954787] Key type .llcrypt unregistered [ 3994.969563] Key type ._llcrypt unregistered [ 4020.456454] alg: No test for adler32 (adler32-zlib) [ 4021.211456] Key type ._llcrypt registered [ 4021.216175] Key type .llcrypt registered [ 4021.366740] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4022.714186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4023.698101] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 4024.001795] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4024.006774] LNet: Accept secure, port 988 [ 4025.703679] Key type lgssc registered [ 4026.750900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4034.539031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.933376] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.994657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4040.268673] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4041.186804] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.304213] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4048.603247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4049.086797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4053.417980] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4061.239703] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4067.461845] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4074.165061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4074.531835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.468877] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4078.641318] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 4085.411876] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4085.815187] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 4104.550252] Lustre: lustre-OST0000: evicting 418558bf-b652-4517-b996-13951bc47966 (at 192.168.203.45@tcp) by administrative request [ 4104.564658] Lustre: Skipped 1 previous similar message [ 4134.882525] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4134.899717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.917837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4136.931859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4136.932288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.942539] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4139.520687] Lustre: server umount lustre-OST0000 complete [ 4146.145414] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4146.155862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.172225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4146.178508] Lustre: Skipped 1 previous similar message [ 4152.295798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4152.303423] Lustre: Skipped 3 previous similar messages [ 4157.407161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4157.408433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4157.431018] Lustre: Skipped 1 previous similar message [ 4157.495493] Lustre: server umount lustre-MDT0000 complete [ 4161.067267] LustreError: 152404:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199894 with bad export cookie 17345291951278548936 [ 4161.078070] LustreError: 152404:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4161.085731] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4161.319915] Lustre: server umount lustre-MDT0001 complete [ 4174.293371] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4178.408507] Key type lgssc unregistered [ 4178.759326] LNet: 156623:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4178.787965] LNet: Removed LNI 192.168.203.145@tcp [ 4179.496297] Key type .llcrypt unregistered [ 4179.501845] Key type ._llcrypt unregistered [ 4244.216693] alg: No test for adler32 (adler32-zlib) [ 4244.971333] Key type ._llcrypt registered [ 4244.973029] Key type .llcrypt registered [ 4245.063491] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4246.129417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4246.478189] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 4246.657269] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4246.660634] LNet: Accept secure, port 988 [ 4248.295149] Key type lgssc registered [ 4249.259853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4255.536051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4256.985516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4257.067666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4261.059302] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4262.375294] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4267.493300] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4268.991706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.318439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4273.344876] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4280.471213] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4285.905907] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4292.082097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4292.439879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4296.687639] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4299.633168] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 4304.245351] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4307.838698] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 4318.177347] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4318.180920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.192400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4318.221860] Lustre: Skipped 1 previous similar message [ 4323.296256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4323.313334] Lustre: Skipped 2 previous similar messages [ 4323.476885] Lustre: server umount lustre-OST0000 complete [ 4328.417573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.418519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4328.435030] Lustre: Skipped 1 previous similar message [ 4333.536477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4333.545466] Lustre: Skipped 2 previous similar messages [ 4338.663365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4338.669756] Lustre: Skipped 1 previous similar message [ 4342.239205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4342.437617] Lustre: server umount lustre-MDT0000 complete [ 4343.788898] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4343.808823] LustreError: Skipped 1 previous similar message [ 4346.600225] LustreError: 159142:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200080 with bad export cookie 66356847937358447 [ 4346.601124] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4346.617331] LustreError: 159142:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4346.893291] Lustre: server umount lustre-MDT0001 complete [ 4358.439520] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4361.696862] Key type lgssc unregistered [ 4361.958672] LNet: 162847:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4361.975119] LNet: Removed LNI 192.168.203.145@tcp [ 4362.867376] Key type .llcrypt unregistered [ 4362.874967] Key type ._llcrypt unregistered [ 4393.158637] alg: No test for adler32 (adler32-zlib) [ 4393.923773] Key type ._llcrypt registered [ 4393.927740] Key type .llcrypt registered [ 4394.097285] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4396.147478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4396.828755] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 4397.117349] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4397.123805] LNet: Accept secure, port 988 [ 4398.888118] Key type lgssc registered [ 4400.550446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4410.453209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4421.006957] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 16:55:52 (1776200152) [ 4438.238503] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4447.446616] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4456.142050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4463.553826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4470.599678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4477.984151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4485.949522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4486.010814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4487.363408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4487.403536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4487.523620] Lustre: lustre-MDT0000: new disk, initializing [ 4487.654575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4487.688601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4492.284561] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4501.978989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4502.056282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4502.141729] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4502.189838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4502.203658] Lustre: Skipped 1 previous similar message [ 4502.335201] Lustre: lustre-MDT0001: new disk, initializing [ 4502.475975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4502.523931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4502.535181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4506.682162] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4516.993772] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4522.843161] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4529.110802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4529.167377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4529.175556] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4529.413489] Lustre: lustre-OST0000: new disk, initializing [ 4529.418255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4529.501704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4533.644642] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4536.394583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4536.408479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4543.889708] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4557.282920] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4557.293061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4557.301549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4557.326079] Lustre: Skipped 1 previous similar message [ 4558.793510] Lustre: server umount lustre-OST0000 complete [ 4563.936839] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4563.944483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.953312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4563.956101] Lustre: Skipped 1 previous similar message [ 4567.520657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4567.523885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4572.641059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4572.644347] Lustre: Skipped 2 previous similar messages [ 4577.249103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4577.388215] Lustre: server umount lustre-MDT0000 complete [ 4578.785232] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4578.797958] LustreError: Skipped 1 previous similar message [ 4581.493425] LustreError: 167274:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200315 with bad export cookie 18414809943681779587 [ 4581.508504] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4581.994762] Lustre: server umount lustre-MDT0001 complete [ 4592.733280] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4595.296313] Key type lgssc unregistered [ 4595.555204] LNet: 171280:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4595.573713] LNet: Removed LNI 192.168.203.145@tcp [ 4596.501440] Key type .llcrypt unregistered [ 4596.503716] Key type ._llcrypt unregistered [ 4621.530906] alg: No test for adler32 (adler32-zlib) [ 4622.286538] Key type ._llcrypt registered [ 4622.288966] Key type .llcrypt registered [ 4622.421530] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4623.896182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4624.434496] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 4624.626136] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4624.630807] LNet: Accept secure, port 988 [ 4626.271121] Key type lgssc registered [ 4627.465094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4635.503399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4636.812617] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4636.859368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4641.695599] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4642.273183] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4647.395229] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.539182] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4654.001756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4654.808071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4663.235221] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4676.283510] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4684.058452] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4691.852147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4692.129736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4697.091904] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4704.660231] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4713.663610] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4719.348420] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4719.377880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.393108] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4719.405592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4719.429626] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.145@tcp (at 0@lo) [ 4719.438328] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 4720.623446] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4720.640936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.656572] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4720.669239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4720.685662] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.145@tcp (at 0@lo) [ 4720.690452] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 4721.680729] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4721.718388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4721.727889] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4721.733467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4721.744496] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.145@tcp (at 0@lo) [ 4721.745232] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 4722.591704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4722.599567] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 4723.534551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4723.556814] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:225 [ 4724.467548] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4724.475913] Lustre: Skipped 2 previous similar messages [ 4724.500229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.523927] Lustre: Skipped 2 previous similar messages [ 4724.528612] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4724.538757] Lustre: Skipped 2 previous similar messages [ 4724.545894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4724.562906] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.145@tcp (at 0@lo) [ 4724.564183] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:257 [ 4724.572728] Lustre: Skipped 2 previous similar messages [ 4725.633843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4725.652680] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:289 [ 4726.497374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4726.516930] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:321 [ 4727.641336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4727.656289] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:353 [ 4728.755783] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4728.761156] Lustre: Skipped 3 previous similar messages [ 4728.772345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.779425] Lustre: Skipped 3 previous similar messages [ 4728.783946] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4728.789222] Lustre: Skipped 3 previous similar messages [ 4728.798241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4728.812407] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.145@tcp (at 0@lo) [ 4728.817994] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:385 [ 4728.819126] Lustre: Skipped 3 previous similar messages [ 4739.044301] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4739.053362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4739.073325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4743.694387] Lustre: server umount lustre-OST0000 complete [ 4749.792511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4749.802932] Lustre: Skipped 2 previous similar messages [ 4754.369379] Lustre: server umount lustre-MDT0000 complete [ 4754.915502] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4754.934135] LustreError: Skipped 1 previous similar message [ 4759.071480] LustreError: 172749:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200492 with bad export cookie 17560368165129765938 [ 4759.075507] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4759.090452] LustreError: 172749:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4759.703550] Lustre: server umount lustre-MDT0001 complete [ 4775.626900] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4779.481181] Key type lgssc unregistered [ 4779.842946] LNet: 177269:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4779.862546] LNet: Removed LNI 192.168.203.145@tcp [ 4781.132974] Key type .llcrypt unregistered [ 4781.138801] Key type ._llcrypt unregistered [ 4806.456164] alg: No test for adler32 (adler32-zlib) [ 4807.211801] Key type ._llcrypt registered [ 4807.218066] Key type .llcrypt registered [ 4807.326657] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4808.051344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4808.872304] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 4809.223751] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4809.228342] LNet: Accept secure, port 988 [ 4810.887244] Key type lgssc registered [ 4811.931252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4819.085139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4820.500974] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4820.578202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4824.831489] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4826.083239] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4831.208494] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4833.750354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.054836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4839.151460] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4848.659276] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4855.141248] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4862.143542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4862.483679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4866.880275] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4870.737290] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:417 [ 4874.958956] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4879.771797] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4881.622559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.640518] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4881.649216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4881.661420] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.145@tcp (at 0@lo) [ 4881.661626] LustreError: 178771:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 4883.948404] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4883.963094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4883.975388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4885.984639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.990358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4886.010601] Lustre: Skipped 1 previous similar message [ 4889.613096] Lustre: server umount lustre-OST0000 complete [ 4901.751046] LustreError: 178771:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 4901.768991] LustreError: 178771:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 4925.317843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4925.756631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4927.783906] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4927.800347] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 4927.814321] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:66 to 0x280000400:97 [ 4930.256317] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 4938.776954] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4939.049852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.074673] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.145@tcp (at 0@lo) [ 4939.085175] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:449 [ 4942.808103] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 4949.106095] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff894847f78000.ost_server_uuid 40 [ 4950.979094] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff894847f78000.ost_server_uuid in FULL state after 0 sec [ 4958.689700] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4958.698810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4958.712329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4958.716942] Lustre: Skipped 1 previous similar message [ 4959.712173] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4960.906961] Lustre: server umount lustre-OST0000 complete [ 4964.834765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4964.837481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4964.855245] Lustre: Skipped 2 previous similar messages [ 4964.879554] Lustre: Skipped 3 previous similar messages [ 4969.961926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4969.968178] Lustre: Skipped 1 previous similar message [ 4970.838818] Lustre: server umount lustre-MDT0000 complete [ 4974.314259] LustreError: 178732:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200708 with bad export cookie 14785969238847096967 [ 4974.330321] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4974.331857] LustreError: 178732:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4974.746921] Lustre: server umount lustre-MDT0001 complete [ 4986.739456] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4989.590505] Key type lgssc unregistered [ 4989.893713] LNet: 184046:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4989.906115] LNet: Removed LNI 192.168.203.145@tcp [ 4990.709992] Key type .llcrypt unregistered [ 4990.723081] Key type ._llcrypt unregistered [ 5022.615636] alg: No test for adler32 (adler32-zlib) [ 5023.383400] Key type ._llcrypt registered [ 5023.393645] Key type .llcrypt registered [ 5023.550644] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5025.626971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5026.192712] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 5026.498552] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5026.507217] LNet: Accept secure, port 988 [ 5028.295570] Key type lgssc registered [ 5029.723940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5040.424809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5041.900786] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5041.966784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5046.418483] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5047.272614] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5052.386081] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5054.671895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5055.067593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5059.652688] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5067.318191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5067.721217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5072.323778] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5072.886758] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 5081.511031] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5081.605551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5081.617240] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5081.853208] Lustre: lustre-OST0001: new disk, initializing [ 5081.867604] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5081.999268] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5087.723692] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5090.447891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5090.458939] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5100.874500] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5110.369165] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5131.748467] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.762053] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5131.762231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5131.778775] Lustre: Skipped 1 previous similar message [ 5136.280617] Lustre: server umount lustre-MDT0000 complete [ 5136.874517] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5136.898475] LustreError: Skipped 3 previous similar messages [ 5140.248799] LustreError: 185808:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200874 with bad export cookie 6835979051477079988 [ 5140.257738] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5140.271378] LustreError: 185808:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5140.609964] Lustre: server umount lustre-MDT0001 complete [ 5149.087332] Lustre: 185091:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200875/real 1776200875] req@0000000070b94b14 x1862481488164160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200882 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 5149.115452] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.128643] Lustre: Skipped 2 previous similar messages [ 5151.199164] Lustre: 189618:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200878/real 1776200878] req@0000000057d0cea9 x1862481488164288/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200884 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5151.272509] Lustre: server umount lustre-OST0000 complete [ 5153.119090] Lustre: 185089:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200879/real 1776200879] req@00000000650200d6 x1862481488164352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200886 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 5155.437876] Lustre: server umount lustre-OST0001 complete [ 5158.664578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5158.755874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5159.799246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5159.900241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5160.980489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5161.071536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5162.054480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5162.158978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5176.967534] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5186.281332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5186.726534] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5186.805402] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5191.225599] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5197.280851] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5197.302605] LustreError: Skipped 1 previous similar message [ 5199.953642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.468589] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5205.661624] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5216.573639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5216.802318] LustreError: 137-5: mylustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5216.984296] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.512727] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:484 to 0x0:513 [ 5222.487195] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5232.955812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5239.027572] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5247.857678] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5252.229832] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5269.484347] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5269.493469] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5269.503508] Lustre: Skipped 3 previous similar messages [ 5269.508955] Lustre: Skipped 5 previous similar messages [ 5274.593345] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5274.608808] Lustre: Skipped 4 previous similar messages [ 5279.716723] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.729912] Lustre: Skipped 3 previous similar messages [ 5282.272315] Lustre: mylustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5282.549440] Lustre: server umount mylustre-MDT0000 complete [ 5284.835534] LustreError: 137-5: mylustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5284.850884] LustreError: Skipped 8 previous similar messages [ 5286.038654] LustreError: 191261:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201019 with bad export cookie 6835979051477082711 [ 5286.052637] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5286.076254] LustreError: 191261:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5296.607256] Lustre: 194983:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201024/real 1776201024] req@000000005a5e6856 x1862481488198080/t0(0) o39->mylustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201030 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5296.645859] Lustre: 194983:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5296.709645] Lustre: server umount mylustre-OST0000 complete [ 5296.713694] Lustre: Skipped 1 previous similar message [ 5296.992409] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.330568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5303.439383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5304.335469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5304.425344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5305.186834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5305.273653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5306.068912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5306.148659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5317.040277] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5325.245092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5325.643169] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5325.648376] Lustre: Skipped 1 previous similar message [ 5329.509874] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5337.043960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5341.878589] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5350.947308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5352.423891] LustreError: 137-5: tfs-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5352.441984] LustreError: Skipped 3 previous similar messages [ 5355.350463] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5355.512730] Lustre: tfs-OST0000: deleting orphan objects from 0x0:484 to 0x0:545 [ 5362.625308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5364.150135] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 5367.049291] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5374.491856] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5378.631369] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5393.890345] LustreError: 11-0: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5393.894078] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5393.902980] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 5393.947650] Lustre: Skipped 3 previous similar messages [ 5395.993342] Lustre: server umount tfs-MDT0000 complete [ 5395.997026] Lustre: Skipped 1 previous similar message [ 5399.061370] LustreError: 197448:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201132 with bad export cookie 6835979051477084321 [ 5399.070572] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5408.223408] Lustre: 200289:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201136/real 1776201136] req@00000000ea9427ab x1862481488223936/t0(0) o39->tfs-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201142 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5408.246969] Lustre: 200289:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5410.212354] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5415.042726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5415.106592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5416.091725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5416.182843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5417.118063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5417.178898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5418.192125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5418.291705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5434.000371] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5442.622459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.160968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5443.164395] Lustre: Skipped 3 previous similar messages [ 5447.458733] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5455.364972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.246835] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5468.653807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5473.202368] Lustre: lustre-OST0000: deleting orphan objects from 0x0:484 to 0x0:577 [ 5474.195252] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5482.528659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5483.882251] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 5486.869324] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5494.499951] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5497.855424] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5508.924382] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 17:14:00 (1776201240) [ 5527.524817] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5527.530664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5527.546634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5527.554922] Lustre: Skipped 3 previous similar messages [ 5530.576167] Lustre: server umount lustre-MDT0000 complete [ 5530.580423] Lustre: Skipped 3 previous similar messages [ 5533.996333] LustreError: 201932:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201267 with bad export cookie 6835979051477085952 [ 5534.000499] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5534.005868] LustreError: 201932:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5534.176229] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5534.195849] LustreError: Skipped 17 previous similar messages [ 5558.239118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5575.046647] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5587.392543] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5598.469195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5604.803550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5610.209459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5616.323617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5622.213819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5622.248779] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 5622.248779] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 5622.248779] [ 5622.262820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 5622.474680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5622.502579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5622.591504] Lustre: lustre-MDT0000: new disk, initializing [ 5622.684166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5622.689437] Lustre: Skipped 3 previous similar messages [ 5622.708735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5626.927854] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5636.019988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5636.094423] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 5636.094423] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 5636.094423] [ 5636.114479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 5636.223301] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5636.263353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5636.267340] Lustre: Skipped 1 previous similar message [ 5636.321316] Lustre: lustre-MDT0001: new disk, initializing [ 5636.440572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5636.451048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5640.262270] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5650.324555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5650.394524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5650.399769] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5650.668425] Lustre: lustre-OST0000: new disk, initializing [ 5650.675241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5651.914438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5651.927364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5655.035818] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5664.016932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5664.085493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5664.093623] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5664.224928] Lustre: lustre-OST0001: new disk, initializing [ 5664.229814] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5666.150518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5666.166944] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5668.397834] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5679.795979] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 17:16:51 (1776201411) [ 5692.280591] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 17:17:04 (1776201424) [ 5707.237459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.261780] Lustre: Skipped 5 previous similar messages [ 5707.276070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5707.284743] Lustre: Skipped 11 previous similar messages [ 5713.077442] Lustre: server umount lustre-OST0000 complete [ 5713.081287] Lustre: Skipped 3 previous similar messages [ 5726.917215] LustreError: 208214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201460 with bad export cookie 6835979051477087471 [ 5726.918331] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5726.931012] LustreError: 208214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5745.165732] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5767.830542] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5782.092378] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5784.231271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5793.337031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5802.354675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5809.985361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5817.817901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5825.127157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5825.262937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.454658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5825.483475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5825.538254] Lustre: lustre-MDT0000: new disk, initializing [ 5825.624107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5829.819513] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5840.630829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5840.715945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5841.200786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5846.654795] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5855.971424] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5861.733601] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5868.084034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5868.182636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5868.189434] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5868.385974] Lustre: lustre-OST0000: new disk, initializing [ 5868.388379] Lustre: Skipped 1 previous similar message [ 5868.391851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5868.405660] Lustre: Skipped 2 previous similar messages [ 5870.400420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5870.411777] Lustre: Skipped 1 previous similar message [ 5870.416618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5873.129567] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5883.056426] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5891.047429] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5891.047679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.070182] Lustre: Skipped 5 previous similar messages [ 5891.078201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5891.083878] Lustre: Skipped 12 previous similar messages [ 5895.560872] Lustre: server umount lustre-OST0000 complete [ 5895.564210] Lustre: Skipped 3 previous similar messages [ 5906.402475] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5906.419574] LustreError: Skipped 10 previous similar messages [ 5908.707703] LustreError: 215426:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201642 with bad export cookie 6835979051477088899 [ 5908.720995] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5919.945638] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5922.357713] Key type lgssc unregistered [ 5922.620973] LNet: 219285:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5922.630951] LNet: Removed LNI 192.168.203.145@tcp [ 5923.452416] Key type .llcrypt unregistered [ 5923.461192] Key type ._llcrypt unregistered [ 5944.674623] alg: No test for adler32 (adler32-zlib) [ 5945.430477] Key type ._llcrypt registered [ 5945.432659] Key type .llcrypt registered [ 5945.533555] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5946.502747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5946.831545] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 5947.040952] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5947.044479] LNet: Accept secure, port 988 [ 5948.711134] Key type lgssc registered [ 5949.610185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5957.248776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5958.700993] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.753345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5962.993378] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5963.747590] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.873784] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5971.564636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5972.014595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5976.859921] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 5984.864814] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5990.435244] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5996.837315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5997.193207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6002.252612] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 6008.430867] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 6010.070791] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6011.113781] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 6013.924367] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6013.928304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6013.940513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6013.970095] Lustre: Skipped 1 previous similar message [ 6015.402670] Lustre: server umount lustre-OST0000 complete [ 6023.136268] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6023.140287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.161643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6023.172696] Lustre: Skipped 1 previous similar message [ 6024.164477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6024.167371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6025.588065] Lustre: server umount lustre-MDT0000 complete [ 6029.280894] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6029.300803] LustreError: Skipped 1 previous similar message [ 6029.569809] LustreError: 220780:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201763 with bad export cookie 13925220148188494609 [ 6029.575575] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6029.581363] LustreError: 220780:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6029.889517] Lustre: server umount lustre-MDT0001 complete [ 6041.445413] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6044.355422] Key type lgssc unregistered [ 6044.633836] LNet: 224512:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6044.644226] LNet: Removed LNI 192.168.203.145@tcp [ 6045.461655] Key type .llcrypt unregistered [ 6045.469301] Key type ._llcrypt unregistered [ 6059.681087] alg: No test for adler32 (adler32-zlib) [ 6060.442746] Key type ._llcrypt registered [ 6060.448780] Key type .llcrypt registered [ 6060.596953] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6061.923696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6062.240164] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 6062.492319] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6062.497604] LNet: Accept secure, port 988 [ 6064.199120] Key type lgssc registered [ 6065.324831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6074.205939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6091.935327] LustreError: 225757:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 1: rc = -5 [ 6122.655205] LustreError: 225757:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 6152.352336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6156.780564] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 6164.023826] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6165.989894] Lustre: server umount lustre-OST0000 complete [ 6180.546590] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6182.845261] Key type lgssc unregistered [ 6183.086663] LNet: 227404:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6183.110655] LNet: Removed LNI 192.168.203.145@tcp [ 6183.764650] Key type .llcrypt unregistered [ 6183.767489] Key type ._llcrypt unregistered [ 6223.530336] alg: No test for adler32 (adler32-zlib) [ 6224.291665] Key type ._llcrypt registered [ 6224.294404] Key type .llcrypt registered [ 6224.390114] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6225.566666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6226.111994] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 6226.421469] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6226.431514] LNet: Accept secure, port 988 [ 6228.111144] Key type lgssc registered [ 6229.178441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6241.777228] loop0: detected capacity change from 0 to 204800000 [ 6241.978941] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6242.004694] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6242.437478] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6242.469668] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6242.558857] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6242.635790] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6242.701163] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6242.765273] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6242.860071] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6242.968861] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6244.302452] loop0: detected capacity change from 0 to 204800000 [ 6244.508628] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6245.425729] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6246.590789] loop0: detected capacity change from 0 to 204800000 [ 6247.051704] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6248.232270] loop0: detected capacity change from 0 to 204800000 [ 6248.438600] print_req_error: 12 callbacks suppressed [ 6248.438612] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6248.472183] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6248.631030] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6248.670334] blk_update_request: operation not supported error, dev loop0, sector 58272 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6248.924060] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6250.063947] loop0: detected capacity change from 0 to 204800000 [ 6250.089951] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6254.332428] loop0: detected capacity change from 0 to 204800000 [ 6254.389245] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6258.240893] loop0: detected capacity change from 0 to 204800000 [ 6258.265784] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6262.075531] loop0: detected capacity change from 0 to 204800000 [ 6262.103086] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6266.242770] loop0: detected capacity change from 0 to 204800000 [ 6266.379421] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6266.438970] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.501195] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6266.518259] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 6266.529618] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6266.544185] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 6266.561310] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6266.575451] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 6266.590056] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6266.601389] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 6266.612758] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 6266.622757] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 6266.632474] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 6266.643243] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 6266.653982] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 6266.668642] Lustre: 230889:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 6278.116383] Lustre: 231168:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202005/real 1776202005] req@000000008da8b374 x1862482746409088/t0(0) o251->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202011 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6278.164523] Lustre: server umount MGS complete [ 6279.144894] loop0: detected capacity change from 0 to 204800000 [ 6279.289079] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.631558] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.722452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6279.762886] LustreError: 231222:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 6280.556948] loop1: detected capacity change from 0 to 204800000 [ 6280.662377] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6280.763680] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6280.841649] Lustre: 231337:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6280.849141] Lustre: 231337:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6280.861852] Lustre: 231337:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6280.868785] Lustre: 231337:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 6280.874756] Lustre: 231337:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 6280.881280] Lustre: 231337:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6280.889907] Lustre: 231337:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 6280.905060] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6281.107519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6281.166049] LustreError: 231337:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0001: Aborting client recovery [ 6282.038696] loop2: detected capacity change from 0 to 204800000 [ 6282.135371] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6282.286987] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6282.297882] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 6282.336287] Lustre: 231421:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6282.343967] Lustre: 231421:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6282.351603] Lustre: 231421:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6282.361473] Lustre: 231421:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6282.369785] Lustre: 231421:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6282.378198] Lustre: 231421:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6282.386819] Lustre: 231421:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6282.488053] Lustre: Found index 0 for lustre-OST0000, updating log [ 6282.580303] LustreError: 231280:0:(lod_dev.c:437:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 6282.585248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6282.605725] LustreError: 231421:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 6283.407172] loop3: detected capacity change from 0 to 204800000 [ 6283.542019] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6283.616622] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6283.624271] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 6283.684686] Lustre: 231508:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6283.695603] Lustre: 231508:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6283.705723] Lustre: 231508:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6283.717601] Lustre: 231508:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6283.730776] Lustre: 231508:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6283.739701] Lustre: 231508:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6283.747760] Lustre: 231508:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6283.772092] Lustre: Found index 1 for lustre-OST0001, updating log [ 6283.892572] LustreError: 231508:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 6284.916747] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 6284.922942] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 6287.817364] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 6287.852523] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 6292.078317] LustreError: 231246:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 6047us: rc = -2 [ 6292.626731] LustreError: 231247:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 370us: rc = -2 [ 6292.659492] LustreError: 231247:0:(mdt_identity.c:112:mdt_identity_do_upcall()) Skipped 20 previous similar messages [ 6305.761988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6305.763610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6305.794268] Lustre: Skipped 2 previous similar messages [ 6305.809952] Lustre: Skipped 3 previous similar messages [ 6310.889439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6310.902118] Lustre: Skipped 3 previous similar messages [ 6311.509951] Lustre: server umount lustre-MDT0000 complete [ 6312.437902] LustreError: 231231:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202046 with bad export cookie 12554501716625421904 [ 6312.445264] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6312.453057] LustreError: 231231:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6315.999925] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.002386] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6316.016031] LustreError: Skipped 1 previous similar message [ 6316.030393] Lustre: Skipped 1 previous similar message [ 6316.034204] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6316.044840] Lustre: Skipped 1 previous similar message [ 6318.052193] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6318.054537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6318.070905] LustreError: Skipped 1 previous similar message [ 6318.076152] Lustre: Skipped 1 previous similar message [ 6318.860795] Lustre: server umount lustre-MDT0001 complete [ 6325.811583] Lustre: server umount lustre-OST0000 complete [ 6332.918275] Lustre: server umount lustre-OST0001 complete [ 6345.083227] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 17:27:56 (1776202076) [ 6378.420282] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6388.209217] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6399.615933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6408.078106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6416.260958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6423.436959] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6431.326031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6431.424851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.757524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6431.803880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6431.929799] Lustre: lustre-MDT0000: new disk, initializing [ 6432.035697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6432.043958] Lustre: Skipped 1 previous similar message [ 6432.069503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6437.423679] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 6447.616322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6447.709636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6447.821230] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6447.853309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6447.857112] Lustre: Skipped 1 previous similar message [ 6447.927426] Lustre: lustre-MDT0001: new disk, initializing [ 6448.031616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6448.078333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6448.097078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6452.451597] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 6461.092472] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6467.297442] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6474.116181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6474.209317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6474.219664] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6474.501138] Lustre: lustre-OST0000: new disk, initializing [ 6474.517147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6474.651650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6480.257807] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 6492.260261] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6498.178764] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6507.788253] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6515.371852] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 6553.576432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.577342] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6553.592210] Lustre: Skipped 1 previous similar message [ 6553.603324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6553.617158] Lustre: Skipped 1 previous similar message [ 6563.812840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6563.821754] Lustre: Skipped 3 previous similar messages [ 6567.391140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6567.503982] Lustre: server umount lustre-OST0000 complete [ 6570.980992] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6570.993170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6584.288122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6584.296941] Lustre: Skipped 6 previous similar messages [ 6585.313946] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6585.435929] Lustre: server umount lustre-MDT0000 complete [ 6589.239203] LustreError: 235680:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202323 with bad export cookie 12554501716625424410 [ 6589.241853] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6589.253126] LustreError: 235680:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6600.188833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.531437] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 6618.079160] Lustre: 241077:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202345/real 1776202345] req@0000000096667768 x1862482746474624/t0(0) o251->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202351 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 6618.147899] Lustre: server umount MGS complete [ 6618.154198] Lustre: Skipped 1 previous similar message [ 6631.437042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.914069] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6631.941529] LustreError: 241713:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 6631.992735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6636.495458] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 6637.027689] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6645.477797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6650.164704] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 6658.584756] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6665.390378] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6672.175040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6672.470401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6672.475171] Lustre: Skipped 1 previous similar message [ 6675.578417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6675.596338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6677.450242] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 6685.103756] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6706.664502] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6706.671286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.677475] Lustre: Skipped 1 previous similar message [ 6706.681601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6709.079166] Lustre: server umount lustre-OST0000 complete [ 6717.407967] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6717.414054] LustreError: Skipped 1 previous similar message [ 6717.419695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6717.435938] Lustre: Skipped 2 previous similar messages [ 6722.049992] LustreError: 241680:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202455 with bad export cookie 12554501716625426167 [ 6722.050621] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6722.059118] LustreError: 241680:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6735.684576] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6739.156057] Key type lgssc unregistered [ 6739.486590] LNet: 245819:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6739.498482] LNet: Removed LNI 192.168.203.145@tcp [ 6740.207710] Key type .llcrypt unregistered [ 6740.211459] Key type ._llcrypt unregistered [ 6774.156978] alg: No test for adler32 (adler32-zlib) [ 6774.910478] Key type ._llcrypt registered [ 6774.918209] Key type .llcrypt registered [ 6775.073862] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6786.677811] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6787.794983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6788.389464] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 6788.571533] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6788.574785] LNet: Accept secure, port 988 [ 6790.223422] Key type lgssc registered [ 6791.227924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6799.751324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6805.873241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6812.058943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6818.695155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6824.937956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6825.006821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6826.235615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6826.268342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6826.357826] Lustre: lustre-MDT0000: new disk, initializing [ 6826.425779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6826.444683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6830.580202] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 6841.468466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6841.595759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6841.798996] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6841.878184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6841.884893] Lustre: Skipped 1 previous similar message [ 6841.966466] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6842.054931] Lustre: lustre-MDT0001: new disk, initializing [ 6842.184232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6842.229349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6842.249410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6847.199584] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 6857.705886] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6863.726349] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6870.044413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6870.111343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6870.117465] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6870.366354] Lustre: lustre-OST0000: new disk, initializing [ 6870.375521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6870.473422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6874.811567] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 6878.749692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6878.758518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6882.758968] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6888.534901] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6900.616878] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6905.710939] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 6955.489434] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6955.499926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6955.500784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6955.515475] Lustre: Skipped 1 previous similar message [ 6955.526347] Lustre: Skipped 1 previous similar message [ 6960.614117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6960.623643] Lustre: Skipped 1 previous similar message [ 6965.740145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6965.750871] Lustre: Skipped 1 previous similar message [ 6969.311152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6969.568153] Lustre: server umount lustre-OST0000 complete [ 6975.456626] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6975.462014] LustreError: Skipped 1 previous similar message [ 6975.464517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6975.477599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6986.208384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6986.211583] Lustre: Skipped 4 previous similar messages [ 6987.231130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6987.326884] Lustre: server umount lustre-MDT0000 complete [ 6990.543529] LustreError: 249898:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202724 with bad export cookie 12571592914577355274 [ 6990.546748] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6990.552647] LustreError: 249898:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6990.898891] Lustre: server umount lustre-MDT0001 complete [ 6999.468315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7004.015284] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7015.391519] Lustre: 255344:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202743/real 1776202743] req@00000000e86f448f x1862483335756096/t0(0) o251->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202749 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7015.493381] Lustre: server umount MGS complete [ 7028.145050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7028.806858] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.827316] LustreError: 255980:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7028.889379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7034.337177] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7034.545718] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7039.457882] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7044.218241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.650326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7049.623086] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7058.502699] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7064.504079] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7071.313486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7071.691987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7076.294743] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7083.836887] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7111.650866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7111.653377] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7111.653662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7111.673668] Lustre: Skipped 2 previous similar messages [ 7117.735140] Lustre: server umount lustre-OST0000 complete [ 7121.893979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7126.496089] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7126.503853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7127.728223] Lustre: server umount lustre-MDT0000 complete [ 7131.742783] LustreError: 255948:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202865 with bad export cookie 12571592914577357073 [ 7131.753969] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7131.758190] LustreError: 255948:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7144.242359] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7147.851487] Key type lgssc unregistered [ 7148.191957] LNet: 260082:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7148.219359] LNet: Removed LNI 192.168.203.145@tcp [ 7149.298666] Key type .llcrypt unregistered [ 7149.307358] Key type ._llcrypt unregistered [ 7179.340892] alg: No test for adler32 (adler32-zlib) [ 7180.108541] Key type ._llcrypt registered [ 7180.115047] Key type .llcrypt registered [ 7180.244201] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7181.105994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7181.703550] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 7181.906919] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7181.911205] LNet: Accept secure, port 988 [ 7183.559149] Key type lgssc registered [ 7184.397290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7191.442861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7192.931898] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7193.027767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7197.399269] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7198.179561] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.296819] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.072567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.515975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7210.420737] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7217.482429] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7222.766617] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7229.103909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7229.390731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7233.377349] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7240.926458] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7247.168695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7247.224128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7247.231513] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7247.359728] Lustre: lustre-OST0001: new disk, initializing [ 7247.363914] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7247.433651] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7250.553905] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7254.594925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7254.601697] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7254.641162] LustreError: 265095:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7255.717513] LustreError: 265095:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7255.727984] LustreError: 265095:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 7256.740996] LustreError: 265099:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7258.798336] LustreError: 265099:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7258.813075] LustreError: 265099:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 7260.731974] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7260.930721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7262.949262] LustreError: 265095:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7262.964699] LustreError: 265095:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [ 7266.350379] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff89484431e000.ost_server_uuid 40 [ 7267.516241] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff89484431e000.ost_server_uuid in FULL state after 0 sec [ 7271.138222] LustreError: 265099:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7271.149335] LustreError: 265099:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [ 7272.924549] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89484431e000.ost_server_uuid 40 [ 7274.652095] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89484431e000.ost_server_uuid in FULL state after 0 sec [ 7290.855246] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7290.880144] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7290.888730] Lustre: Skipped 1 previous similar message [ 7291.813905] Lustre: lustre-OST0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 7295.869140] Lustre: server umount lustre-OST0001 complete [ 7295.971479] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7295.997605] LustreError: Skipped 1 previous similar message [ 7301.106580] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7301.118993] LustreError: Skipped 2 previous similar messages [ 7303.138205] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7303.149552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7303.162943] Lustre: Skipped 1 previous similar message [ 7303.175550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7304.160552] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7304.170098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7306.213765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7306.227179] Lustre: Skipped 2 previous similar messages [ 7308.021347] Lustre: server umount lustre-OST0000 complete [ 7311.335116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.343301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.344633] Lustre: Skipped 1 previous similar message [ 7311.359431] Lustre: Skipped 1 previous similar message [ 7317.626554] Lustre: server umount lustre-MDT0000 complete [ 7320.819308] LustreError: 261642:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203054 with bad export cookie 18042944474754206132 [ 7320.836304] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7320.836584] LustreError: 261642:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7321.228883] Lustre: server umount lustre-MDT0001 complete [ 7332.398592] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7334.925179] Key type lgssc unregistered [ 7335.193500] LNet: 267317:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7335.207437] LNet: Removed LNI 192.168.203.145@tcp [ 7336.055837] Key type .llcrypt unregistered [ 7336.062812] Key type ._llcrypt unregistered [ 7372.981429] alg: No test for adler32 (adler32-zlib) [ 7373.740378] Key type ._llcrypt registered [ 7373.746622] Key type .llcrypt registered [ 7373.870491] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7375.059709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7375.877295] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 7376.093428] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7376.097481] LNet: Accept secure, port 988 [ 7377.767722] Key type lgssc registered [ 7378.902763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7387.243806] loop0: detected capacity change from 0 to 1099511627776 [ 7387.780661] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 7387.790332] *** VALIDATE xfs *** [ 7387.804381] XFS (loop0): Mounting V5 Filesystem [ 7387.829507] XFS (loop0): Ending clean mount [ 7389.387801] loop1: detected capacity change from 0 to 18691697672192 [ 7484.378291] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7501.197656] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 7510.038025] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7519.863031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7526.801336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7533.331776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7540.663711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7547.770856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7547.840874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7549.230695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7549.258929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7549.354201] Lustre: lustre-MDT0000: new disk, initializing [ 7549.484886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7549.511611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7554.494949] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7564.938721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7565.001895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7565.108723] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7565.173858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7565.183664] Lustre: Skipped 1 previous similar message [ 7565.297502] Lustre: lustre-MDT0001: new disk, initializing [ 7565.393535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7565.450574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7565.463207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7570.251620] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7580.155367] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7586.831806] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7593.644438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7593.744748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7593.752829] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7594.044619] Lustre: lustre-OST0000: new disk, initializing [ 7594.048239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7594.122568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7598.955831] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7600.745741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7600.755359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7609.840494] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7615.353278] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7615.599331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7620.223371] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7620.389798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7626.720557] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7626.735679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7626.738312] LustreError: Skipped 1 previous similar message [ 7626.739276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7626.761341] Lustre: Skipped 1 previous similar message [ 7628.635389] Lustre: server umount lustre-OST0000 complete [ 7636.973108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7636.979890] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7636.999695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7636.999695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7636.999702] Lustre: Skipped 1 previous similar message [ 7642.081500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7642.086629] Lustre: Skipped 1 previous similar message [ 7647.199595] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7647.313411] Lustre: server umount lustre-MDT0000 complete [ 7648.226144] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7648.237388] LustreError: Skipped 1 previous similar message [ 7651.435657] LustreError: 273128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203385 with bad export cookie 7597683569980393455 [ 7651.448167] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7651.454650] LustreError: 273128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7651.819203] Lustre: server umount lustre-MDT0001 complete [ 7660.710396] XFS (loop0): Unmounting Filesystem [ 7665.164442] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 17:49:57 (1776203397) [ 7672.103814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.677904] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7672.775547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.760589] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7677.922165] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7683.048150] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.779912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7687.214516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7692.109395] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7699.327571] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7705.170437] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7711.607486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7711.939226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7716.245717] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7723.148993] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7739.874965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7739.876685] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7739.897764] Lustre: Skipped 1 previous similar message [ 7739.899326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7739.899346] Lustre: Skipped 2 previous similar messages [ 7744.466574] Lustre: server umount lustre-OST0000 complete [ 7748.577848] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7748.586686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7748.601596] Lustre: Skipped 1 previous similar message [ 7748.615721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7748.621670] Lustre: Skipped 1 previous similar message [ 7754.684719] Lustre: server umount lustre-MDT0000 complete [ 7755.238601] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7755.278731] LustreError: Skipped 1 previous similar message [ 7758.307199] LustreError: 276957:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203492 with bad export cookie 7597683569980394302 [ 7758.329517] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7771.117215] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7774.213669] Key type lgssc unregistered [ 7774.530636] LNet: 280730:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7774.542207] LNet: Removed LNI 192.168.203.145@tcp [ 7775.332383] Key type .llcrypt unregistered [ 7775.335295] Key type ._llcrypt unregistered [ 7803.179546] alg: No test for adler32 (adler32-zlib) [ 7803.938785] Key type ._llcrypt registered [ 7803.943782] Key type .llcrypt registered [ 7804.074350] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7804.946266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7805.840586] Lustre: Lustre: Build Version: 2.15.8_4_g2ff0e3c [ 7806.057361] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7806.060898] LNet: Accept secure, port 988 [ 7807.751727] Key type lgssc registered [ 7808.923400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7816.662386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.960761] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7818.023736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7822.107141] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7823.329192] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7828.453147] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7829.532667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7829.941088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7833.851826] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7840.806707] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7846.405735] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7853.819875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7854.354906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7860.027154] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7868.896697] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7879.139975] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7879.168458] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7879.180883] Lustre: Skipped 1 previous similar message [ 7881.107248] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 7883.276925] Lustre: server umount lustre-MDT0001 complete [ 7884.257398] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7884.271743] LustreError: Skipped 1 previous similar message [ 7886.572176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7889.380288] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7889.398295] LustreError: Skipped 2 previous similar messages [ 7893.389739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.478617] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7893.482571] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7893.532545] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7893.924363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7898.649513] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 7899.117818] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7899.123381] LustreError: 285971:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 7899.131366] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 7899.139095] LustreError: 285971:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7899.139116] LustreError: 285971:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7899.139944] LustreError: 282360:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7899.188753] Lustre: Skipped 1 previous similar message [ 7899.207915] LustreError: 285975:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -17 [ 7899.220386] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 7899.220386] [ 8206.305605] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8206.323398] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8206.332513] Lustre: Skipped 1 previous similar message [ 8206.336868] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8206.339932] Lustre: Skipped 1 previous similar message [ 8210.023948] Lustre: server umount lustre-MDT0001 complete [ 8211.431807] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8211.442448] LustreError: Skipped 2 previous similar messages [ 8212.919836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8221.038353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.210470] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8221.216765] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8221.265598] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8221.552756] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 8226.282404] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 8226.809200] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 8226.825690] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8227.323268] LustreError: 294791:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 8227.334809] LustreError: 294791:0:(genops.c:522:class_register_device()) Skipped 1 previous similar message [ 8227.347034] LustreError: 294791:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8227.356757] LustreError: 294791:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8227.367640] LustreError: 282360:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 8227.371544] LustreError: 282360:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 8227.436552] LustreError: 294799:0:(obd_config.c:680:class_setup()) Device 8 already setup (type osp) [ 8227.443142] LustreError: 294799:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -17 [ 8227.455307] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.145@tcp [ 8227.455307] [ 8533.986784] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8533.988252] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8533.988758] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8533.997773] Lustre: Skipped 2 previous similar messages [ 8534.023266] Lustre: Skipped 2 previous similar messages [ 8537.833938] Lustre: server umount lustre-MDT0001 complete [ 8539.121130] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8539.143260] LustreError: Skipped 5 previous similar messages [ 8540.559522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8547.590817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8547.676882] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8547.684286] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8547.722771] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8547.965765] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 8552.140956] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 8553.447135] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8553.467478] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 8553.472621] Lustre: Skipped 1 previous similar message [ 8556.047694] LustreError: 303687:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 8556.057250] LustreError: 303687:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8556.071824] LustreError: 303687:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8556.081964] LustreError: 282360:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 8556.088031] LustreError: 282360:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 8556.185918] LustreError: 303698:0:(lod_lov.c:228:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 8556.210122] LustreError: 303698:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -114 [ 8556.223361] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 8556.223361] [ 8875.489357] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8875.501908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8875.516187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8875.521062] Lustre: Skipped 1 previous similar message [ 8878.904201] Lustre: server umount lustre-MDT0000 complete [ 8881.121718] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8881.131487] LustreError: Skipped 6 previous similar messages [ 8882.207810] LustreError: 282349:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204616 with bad export cookie 12686959157805798930 [ 8882.211368] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8882.225657] LustreError: 282349:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8886.243729] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.259367] Lustre: Skipped 2 previous similar messages [ 8886.267415] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8886.277924] Lustre: Skipped 2 previous similar messages [ 8888.521561] Lustre: server umount lustre-MDT0001 complete [ 8902.639630] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 8911.996872] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8921.477668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8928.422384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8934.427613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8940.384572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8946.098981] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 18:11:18 (1776204678) [ 8952.385820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8952.457601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8952.693941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8952.716873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8952.830828] Lustre: lustre-MDT0000: new disk, initializing [ 8952.945334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8952.977354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8957.335917] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 8967.894753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8967.969588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8968.105937] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8968.172979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8968.178636] Lustre: Skipped 1 previous similar message [ 8968.277967] Lustre: lustre-MDT0001: new disk, initializing [ 8968.400116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8968.439598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8968.471075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8973.294420] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 8983.777816] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8989.512205] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8997.685831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8997.779324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8997.796512] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8998.161246] Lustre: lustre-OST0000: new disk, initializing [ 8998.166227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8998.265927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9003.167714] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all 8 [ 9007.664616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9007.676524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9012.315744] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9019.880395] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9019.894896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9019.914712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9019.924493] Lustre: Skipped 2 previous similar messages [ 9021.112229] Lustre: server umount lustre-MDT0000 complete [ 9021.117356] Lustre: Skipped 1 previous similar message [ 9023.464620] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9023.490094] LustreError: Skipped 5 previous similar messages [ 9024.735381] LustreError: 315420:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 9024.738134] LustreError: 317322:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204758 with bad export cookie 12686959157805800848 [ 9024.746275] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9024.763171] LustreError: 315420:0:(llog_cat.c:739:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 9024.768417] LustreError: 317322:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9024.795941] LustreError: 315420:0:(llog_cat.c:775:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 9035.747282] Lustre: 317810:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204763/real 1776204763] req@00000000a05ad24e x1862484402297856/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776204769 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9060.793905] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 9069.843738] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9082.848911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9091.506029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9098.625119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9105.471883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9107.403631] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8864 sec ========== 18:13:59 (1776204839) [ 9146.645792] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 9148.952494] Key type lgssc unregistered [ 9149.147813] LNet: 322919:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9149.160421] LNet: Removed LNI 192.168.203.145@tcp [ 9149.899293] Key type .llcrypt unregistered [ 9149.901224] Key type ._llcrypt unregistered