[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 542418536 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.002374] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.007572] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008039] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010003] pid_max: default: 32768 minimum: 301 [ 0.011157] LSM: Security Framework initializing [ 0.012069] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.014063] *** VALIDATE selinux *** [ 0.022378] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027063] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030154] *** VALIDATE tmpfs *** [ 0.032008] *** VALIDATE proc *** [ 0.033291] *** VALIDATE cgroup *** [ 0.034015] *** VALIDATE cgroup2 *** [ 0.036144] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039046] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043896] debug: unmapping init [mem 0xffffffffb5059000-0xffffffffb5060fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046818] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048018] ... bit width: 48 [ 0.049010] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054362] rcu: Hierarchical SRCU implementation. [ 0.056886] smp: Bringing up secondary CPUs ... [ 0.057623] x86: Booting SMP configuration: [ 0.058025] .... node #0, CPUs: #1 #2 #3 [ 0.061540] smp: Brought up 1 node, 4 CPUs [ 0.063033] smpboot: Max logical packages: 1 [ 0.064022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.234140] node 0 deferred pages initialised in 167ms [ 0.237392] devtmpfs: initialized [ 0.239309] x86/mm: Memory block size: 128MB [ 0.243425] gcov: version magic: 0x41383552 [ 0.246419] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.247113] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.248446] pinctrl core: initialized pinctrl subsystem [ 0.249227] [ 0.249634] ************************************************************* [ 0.250015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.251017] ** ** [ 0.252013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.253015] ** ** [ 0.254015] ** This means that this kernel is built to expose internal ** [ 0.255016] ** IOMMU data structures, which may compromise security on ** [ 0.256013] ** your system. ** [ 0.257018] ** ** [ 0.258013] ** If you see this message and you are not debugging the ** [ 0.259015] ** kernel, report this immediately to your vendor! ** [ 0.260014] ** ** [ 0.261014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262015] ************************************************************* [ 0.263973] NET: Registered protocol family 16 [ 0.265719] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.269101] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.271097] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.275048] cpuidle: using governor menu [ 0.276721] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.278394] PCI: Using configuration type 1 for base access [ 0.280094] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.286111] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.287038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.289101] cryptd: max_cpu_qlen set to 1000 [ 0.291781] ACPI: Added _OSI(Module Device) [ 0.293016] ACPI: Added _OSI(Processor Device) [ 0.294013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.295015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.300883] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.308522] ACPI: Interpreter enabled [ 0.310072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.312012] ACPI: Using IOAPIC for interrupt routing [ 0.313122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.316504] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.327302] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.329055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.331021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.334095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.338533] acpiphp: Slot [2] registered [ 0.340144] acpiphp: Slot [5] registered [ 0.341120] acpiphp: Slot [6] registered [ 0.342118] acpiphp: Slot [7] registered [ 0.343158] acpiphp: Slot [8] registered [ 0.344112] acpiphp: Slot [9] registered [ 0.345191] acpiphp: Slot [10] registered [ 0.347135] acpiphp: Slot [3] registered [ 0.348081] acpiphp: Slot [4] registered [ 0.349127] acpiphp: Slot [11] registered [ 0.351152] acpiphp: Slot [12] registered [ 0.352121] acpiphp: Slot [13] registered [ 0.353095] acpiphp: Slot [14] registered [ 0.355109] acpiphp: Slot [15] registered [ 0.356083] acpiphp: Slot [16] registered [ 0.357083] acpiphp: Slot [17] registered [ 0.358112] acpiphp: Slot [18] registered [ 0.359097] acpiphp: Slot [19] registered [ 0.360080] acpiphp: Slot [20] registered [ 0.361096] acpiphp: Slot [21] registered [ 0.363112] acpiphp: Slot [22] registered [ 0.364090] acpiphp: Slot [23] registered [ 0.365118] acpiphp: Slot [24] registered [ 0.367135] acpiphp: Slot [25] registered [ 0.368113] acpiphp: Slot [26] registered [ 0.369108] acpiphp: Slot [27] registered [ 0.370177] acpiphp: Slot [28] registered [ 0.372102] acpiphp: Slot [29] registered [ 0.373109] acpiphp: Slot [30] registered [ 0.374073] acpiphp: Slot [31] registered [ 0.376055] PCI host bridge to bus 0000:00 [ 0.377021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.379027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.381026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.383024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.385023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.388027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.389000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.389953] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.392303] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.400962] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.405055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.407025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.410038] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.413023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.415623] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.417753] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.420042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.422856] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.425807] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.435023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.439000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.444109] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.469021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.485023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.526022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.543151] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.558016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.565018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.592028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.604940] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.614025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.622020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.646023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.659148] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.668020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.678025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.706026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.716905] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.731017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.735016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.763020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.772084] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.786022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.797017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.816022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.826584] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.828372] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.831381] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.833429] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.836235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.840212] iommu: Default domain type: Passthrough [ 0.843392] SCSI subsystem initialized [ 0.845135] ACPI: bus type USB registered [ 0.846095] usbcore: registered new interface driver usbfs [ 0.848109] usbcore: registered new interface driver hub [ 0.850167] usbcore: registered new device driver usb [ 0.852279] pps_core: LinuxPPS API ver. 1 registered [ 0.854013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.857069] PTP clock support registered [ 0.859098] EDAC MC: Ver: 3.0.0 [ 0.860563] PCI: Using ACPI for IRQ routing [ 0.862885] NetLabel: Initializing [ 0.864012] NetLabel: domain hash size = 128 [ 0.866014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.868084] NetLabel: unlabeled traffic allowed by default [ 0.870170] vgaarb: loaded [ 0.872321] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.874014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.881974] clocksource: Switched to clocksource kvm-clock [ 1.004825] VFS: Disk quotas dquot_6.6.0 [ 1.006267] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.011172] *** VALIDATE ramfs *** [ 1.012403] *** VALIDATE hugetlbfs *** [ 1.016135] pnp: PnP ACPI init [ 1.018772] pnp: PnP ACPI: found 6 devices [ 1.036139] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.039245] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.041339] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.043247] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.045621] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.047903] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.050421] NET: Registered protocol family 2 [ 1.052818] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.057322] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.060804] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.065913] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.144034] TCP: Hash tables configured (established 65536 bind 65536) [ 1.146168] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.149275] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.151820] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.154506] NET: Registered protocol family 1 [ 1.158551] RPC: Registered named UNIX socket transport module. [ 1.160625] RPC: Registered udp transport module. [ 1.162351] RPC: Registered tcp transport module. [ 1.164120] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.166499] NET: Registered protocol family 44 [ 1.168222] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.170482] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.171994] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.173542] PCI: CLS 0 bytes, default 64 [ 1.174920] Unpacking initramfs... [ 2.553462] debug: unmapping init [mem 0xffff94e17cc54000-0xffff94e17ffbffff] [ 2.557339] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.559550] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.562451] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.093423] Initialise system trusted keyrings [ 3.094823] Key type blacklist registered [ 3.096643] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.109795] zbud: loaded [ 3.113579] *** VALIDATE nfs *** [ 3.114610] *** VALIDATE nfs4 *** [ 3.116540] pstore: using deflate compression [ 3.121228] Platform Keyring initialized [ 3.232476] NET: Registered protocol family 38 [ 3.234293] Key type asymmetric registered [ 3.235771] Asymmetric key parser 'x509' registered [ 3.237472] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.240433] io scheduler mq-deadline registered [ 3.242038] io scheduler kyber registered [ 3.243772] io scheduler bfq registered [ 3.245656] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.248456] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.250901] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.253557] ACPI: Power Button [PWRF] [ 3.258608] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.264934] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.286445] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.301746] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.326231] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.362211] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.395331] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.401618] Non-volatile memory driver v1.3 [ 3.403762] Linux agpgart interface v0.103 [ 3.448989] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.451818] vda: detected capacity change from 0 to 34836480 [ 3.481390] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.485143] vdb: detected capacity change from 0 to 1073741824 [ 3.522299] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.524738] vdc: detected capacity change from 0 to 2621440000 [ 3.543184] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.546331] vdd: detected capacity change from 0 to 2621440000 [ 3.566447] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.569693] vde: detected capacity change from 0 to 4294967296 [ 3.591597] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.594337] vdf: detected capacity change from 0 to 4294967296 [ 3.602616] libphy: Fixed MDIO Bus: probed [ 3.609249] usbcore: registered new interface driver usbserial_generic [ 3.611648] usbserial: USB Serial support registered for generic [ 3.613910] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.618153] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.619749] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.622468] mousedev: PS/2 mouse device common for all mice [ 3.627024] rtc_cmos 00:05: RTC can wake from S4 [ 3.629401] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.634800] rtc_cmos 00:05: registered as rtc0 [ 3.639213] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.641893] intel_pstate: CPU model not supported [ 3.645350] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.647760] hid: raw HID events driver (C) Jiri Kosina [ 3.652742] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.653366] usbcore: registered new interface driver usbhid [ 3.700042] usbhid: USB HID core driver [ 3.702146] drop_monitor: Initializing network drop monitor service [ 3.703652] Initializing XFRM netlink socket [ 3.705036] NET: Registered protocol family 10 [ 3.707645] Segment Routing with IPv6 [ 3.709315] NET: Registered protocol family 17 [ 3.712821] mpls_gso: MPLS GSO support [ 3.719439] RAS: Correctable Errors collector initialized. [ 3.721598] AVX version of gcm_enc/dec engaged. [ 3.723263] AES CTR mode by8 optimization enabled [ 3.798287] sched_clock: Marking stable (3798241126, 0)->(4705464540, -907223414) [ 3.801977] registered taskstats version 1 [ 3.804127] Loading compiled-in X.509 certificates [ 3.806381] zswap: loaded using pool lzo/zbud [ 3.829647] Key type big_key registered [ 3.923418] Key type encrypted registered [ 3.987123] ima: No TPM chip found, activating TPM-bypass! [ 3.988802] ima: Allocated hash algorithm: sha1 [ 3.990600] ima: No architecture policies found [ 3.992131] evm: Initialising EVM extended attributes: [ 3.994178] evm: security.selinux [ 3.995483] evm: security.ima [ 3.996735] evm: security.capability [ 3.998293] evm: HMAC attrs: 0x1 [ 4.001243] rtc_cmos 00:05: setting system clock to 2026-04-14 18:43:59 UTC (1776192239) [ 4.007433] debug: unmapping init [mem 0xffffffffb6003000-0xffffffffb61fffff] [ 4.010710] debug: unmapping init [mem 0xffffffffb4d82000-0xffffffffb5058fff] [ 4.022169] Write protecting the kernel read-only data: 28672k [ 4.025042] debug: unmapping init [mem 0xffffffffb3403000-0xffffffffb35fffff] [ 4.026992] debug: unmapping init [mem 0xffffffffb3d14000-0xffffffffb3dfffff] [ 4.063378] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.072544] systemd[1]: Detected virtualization kvm. [ 4.074778] systemd[1]: Detected architecture x86-64. [ 4.076863] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.104890] systemd[1]: No hostname configured. [ 4.107677] systemd[1]: Set hostname to . [ 4.109229] random: systemd: uninitialized urandom read (16 bytes read) [ 4.112130] systemd[1]: Initializing machine ID from random generator. [ 4.177855] random: ln: uninitialized urandom read (6 bytes read) [ 4.385412] random: systemd: uninitialized urandom read (16 bytes read) [ 4.388285] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.399220] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.404991] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.194448] device-mapper: uevent: version 1.0.3 [ 5.196498] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.105139] virtio_net virtio0 ens2: renamed from eth0 [ 6.171557] random: fast init done [ 6.299547] scsi host0: ata_piix [ 6.336590] scsi host1: ata_piix [ 6.337808] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.340280] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.505425] random: crng init done [ 11.507074] random: 7 urandom warning(s) missed due to ratelimiting [ 11.956745] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.663095] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.387173] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.715988] SELinux: Disabled at runtime. [ 14.791460] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.799487] systemd[1]: Detected virtualization kvm. [ 14.801235] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.491516] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.495327] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.501812] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.506028] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.510077] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.517285] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.530907] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ 15.670327] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 16.459943] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.797157] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.804893] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.084545] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.154558] EDAC sbridge: Ver: 1.1.2 [ 19.090381] Key type dns_resolver registered [ 19.386759] NFS: Registering the id_resolver key type [ 19.388901] Key type id_resolver registered [ 19.390479] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg245-server login: [ 44.761699] libcfs: loading out-of-tree module taints kernel. [ 44.784885] alg: No test for adler32 (adler32-zlib) [ 45.537358] Key type ._llcrypt registered [ 45.538910] Key type .llcrypt registered [ 45.588145] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 60.066889] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 61.775504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.907339] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 64.073885] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 64.079601] LNet: Accept secure, port 988 [ 65.992802] Key type lgssc registered [ 68.141114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.575555] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 91.405217] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.647829] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 109.991155] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.022511] hrtimer: interrupt took 11707949 ns [ 118.324895] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.770531] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 150.551949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 150.677532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.234265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 152.310165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 152.414598] Lustre: lustre-MDT0000: new disk, initializing [ 152.640884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.699439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 158.471349] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 172.548057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.651530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.851901] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 172.895223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 172.908977] Lustre: Skipped 1 previous similar message [ 172.999702] Lustre: lustre-MDT0001: new disk, initializing [ 173.077411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.118482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.124552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.590662] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 191.457559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.550265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.560568] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 191.761804] Lustre: lustre-OST0000: new disk, initializing [ 191.767213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 191.831520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.245794] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 197.667930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 197.688774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 209.344527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.448583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.459375] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 209.543740] Lustre: lustre-OST0001: new disk, initializing [ 209.548398] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 209.627618] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.677672] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 217.136536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 217.150400] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 224.902173] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 230.356396] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 239.466749] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing check_logdir /tmp/testlogs/ [ 244.828403] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing yml_node [ 249.349613] Lustre: DEBUG MARKER: Client: 2.15.8.10 [ 251.845358] Lustre: DEBUG MARKER: MDS: 2.15.8.10 [ 254.226346] Lustre: DEBUG MARKER: OSS: 2.15.8.10 [ 255.934769] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 14:48:09 EDT 2026 [ 264.461577] Lustre: DEBUG MARKER: excepting tests: 32newtarball 110 [ 265.941141] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 283.615887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.617052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 283.630166] Lustre: Skipped 1 previous similar message [ 283.642671] Lustre: Skipped 3 previous similar messages [ 288.094884] Lustre: server umount lustre-MDT0000 complete [ 288.736268] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 288.747352] LustreError: Skipped 3 previous similar messages [ 291.974665] LustreError: 6249:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192527 with bad export cookie 12600466244661804339 [ 291.978481] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 291.991142] LustreError: 6249:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 293.857094] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.864665] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.864994] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 293.879575] LustreError: Skipped 1 previous similar message [ 293.910717] Lustre: Skipped 3 previous similar messages [ 296.869118] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.869485] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 296.909617] Lustre: Skipped 2 previous similar messages [ 298.367595] Lustre: server umount lustre-MDT0001 complete [ 302.156454] Lustre: server umount lustre-OST0000 complete [ 305.971490] Lustre: server umount lustre-OST0001 complete [ 312.332689] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 323.632788] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 336.434889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 341.918342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 349.733280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 357.136709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 363.885524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 363.985539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.258908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 364.294991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 364.399181] Lustre: lustre-MDT0000: new disk, initializing [ 364.510361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.528827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 369.393259] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 378.938901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 379.012208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.073636] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 379.107491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 379.113287] Lustre: Skipped 1 previous similar message [ 379.174719] Lustre: lustre-MDT0001: new disk, initializing [ 379.256493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 379.294191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 379.311764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 383.306978] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 391.751937] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 398.053392] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 405.692814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 405.820964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 405.825442] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 406.021901] Lustre: lustre-OST0000: new disk, initializing [ 406.024752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 406.078509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.982445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 407.998229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 411.135109] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 422.644558] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 427.276693] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 427.562713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 432.906809] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 433.251122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 438.751970] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 438.756426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.758540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 438.768495] Lustre: Skipped 1 previous similar message [ 438.771266] Lustre: Skipped 2 previous similar messages [ 442.306465] Lustre: server umount lustre-OST0000 complete [ 448.470200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.496842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 448.498817] Lustre: Skipped 1 previous similar message [ 454.883616] Lustre: server umount lustre-MDT0000 complete [ 459.745294] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 459.782695] LustreError: Skipped 4 previous similar messages [ 460.171951] LustreError: 14338:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192695 with bad export cookie 12600466244661805788 [ 460.201729] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 467.022743] Lustre: DEBUG MARKER: == conf-sanity test 154: expand .. on rename after MDT backup restore ========================================================== 14:51:40 (1776192700) [ 483.894258] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 496.916875] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 508.648840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 515.341777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 521.824919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 528.517843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 544.241488] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 554.790807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 554.892879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.206236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 555.251092] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 555.549969] Lustre: lustre-MDT0000: new disk, initializing [ 555.870798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.934210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 562.991857] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 579.072809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 579.191649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.370932] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 579.901943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 585.779084] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 598.573458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 598.665678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 598.672697] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 600.829190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 603.601624] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 616.668962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 616.772642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 616.779323] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 621.474077] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 633.311209] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 636.981751] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 656.865399] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 656.880541] LustreError: Skipped 1 previous similar message [ 656.885469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 656.908122] Lustre: Skipped 1 previous similar message [ 656.919791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 656.922178] Lustre: Skipped 5 previous similar messages [ 662.141304] Lustre: server umount lustre-MDT0000 complete [ 662.143118] Lustre: Skipped 1 previous similar message [ 664.555091] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.573976] LustreError: Skipped 3 previous similar messages [ 666.509502] LustreError: 21940:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192902 with bad export cookie 12600466244661806649 [ 666.512336] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 666.521702] LustreError: 21940:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 676.703154] Lustre: 3363:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192905/real 1776192905] req@000000009fad37fb x1862472616217920/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192912 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 676.733521] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.762093] Lustre: Skipped 3 previous similar messages [ 678.367272] Lustre: 25841:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192907/real 1776192907] req@000000002c9f5ff8 x1862472616218176/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192913 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 679.841179] Lustre: 3362:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192908/real 1776192908] req@000000007f10a538 x1862472616218304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192915 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 679.883831] Lustre: 3362:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 681.951131] Lustre: 3362:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192910/real 1776192910] req@00000000d13e40c2 x1862472616218368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192917 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 697.242600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 707.141333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 708.410854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 717.875657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 728.606314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 729.944827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 751.061724] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 763.264268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 763.278091] Lustre: lustre-MDT0000: reset Object Index mappings [ 763.900385] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 764.035646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 764.045678] Lustre: Skipped 3 previous similar messages [ 769.030616] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 779.922942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 779.947972] Lustre: lustre-MDT0001: reset Object Index mappings [ 786.306562] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 796.116736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 802.606879] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 810.922043] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 810.943319] LustreError: Skipped 4 previous similar messages [ 812.343599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 814.712938] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 814.719042] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 819.572559] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 829.275541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 834.337847] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 845.292306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 845.297932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 845.329119] Lustre: Skipped 3 previous similar messages [ 845.345620] Lustre: Skipped 7 previous similar messages [ 849.610167] Lustre: server umount lustre-MDT0000 complete [ 849.612755] Lustre: Skipped 3 previous similar messages [ 850.400116] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 853.291806] LustreError: 29392:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193088 with bad export cookie 12600466244661813264 [ 853.293155] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 853.316371] LustreError: 29392:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 862.623257] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193091/real 1776193091] req@00000000d995933e x1862472616244480/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776193098 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 890.029343] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:58:43 (1776193123) [ 911.705411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.516033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 912.528837] Lustre: Skipped 3 previous similar messages [ 917.990139] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 917.999237] LustreError: Skipped 4 previous similar messages [ 918.345502] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 928.922733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.433632] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 942.692463] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 949.078686] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 951.786861] Lustre: Failing over lustre-MDT0000 [ 952.057351] Lustre: server umount lustre-MDT0000 complete [ 952.070494] Lustre: Skipped 3 previous similar messages [ 953.831190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.850538] Lustre: Skipped 2 previous similar messages [ 960.992177] Lustre: 3362:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193189/real 1776193189] req@00000000ee717b56 x1862472616253568/t0(0) o400->MGC192.168.202.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776193196 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 961.011355] Lustre: 3362:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 961.037483] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 972.716614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.403816] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0xaeddd94a155e2821 to 0xaeddd94a155e2a66 [ 978.438670] Lustre: MGC192.168.202.145@tcp: Connection restored to (at 0@lo) [ 978.958678] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 979.966156] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 979.977294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 980.025839] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 985.220381] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 996.935531] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 1004.522216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.531753] Lustre: Skipped 1 previous similar message [ 1012.873150] LustreError: 37814:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193248 with bad export cookie 12600466244661815910 [ 1012.880949] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.892767] LustreError: 37814:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1023.286479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.927609] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 1041.751441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.456306] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 1058.981506] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1066.818634] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1069.338217] Lustre: Failing over lustre-MDT0000 [ 1069.395719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.410078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1069.417763] Lustre: Skipped 3 previous similar messages [ 1069.438772] Lustre: Skipped 3 previous similar messages [ 1072.486738] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.494125] LustreError: Skipped 27 previous similar messages [ 1078.563174] Lustre: 3363:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193307/real 1776193307] req@000000005cbae31f x1862472616271744/t0(0) o400->MGC192.168.202.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776193314 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1078.623179] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1091.428577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.226886] Lustre: Evicted from MGS (at 192.168.202.145@tcp) after server handle changed from 0xaeddd94a155e2c0a to 0xaeddd94a155e2e4f [ 1096.271962] Lustre: MGC192.168.202.145@tcp: Connection restored to (at 0@lo) [ 1096.281533] Lustre: Skipped 1 previous similar message [ 1096.688318] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1101.737519] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1101.759230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1101.778772] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1102.080805] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 1115.184806] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 1122.283658] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1126.261831] Lustre: server umount lustre-MDT0000 complete [ 1126.263810] Lustre: Skipped 3 previous similar messages [ 1131.963790] LustreError: 40196:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193367 with bad export cookie 12600466244661816911 [ 1131.970338] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1131.977056] LustreError: 40196:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1143.153852] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:02:56 (1776193376) [ 1162.691667] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 1173.098357] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1185.560876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1193.709554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1201.107971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1208.621916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1218.089318] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1229.796496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1229.942918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.345933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1230.408906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1230.411682] Lustre: Skipped 4 previous similar messages [ 1230.469781] Lustre: lustre-MDT0000: new disk, initializing [ 1230.473129] Lustre: Skipped 3 previous similar messages [ 1230.588200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.592400] Lustre: Skipped 5 previous similar messages [ 1230.602915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1230.606471] Lustre: Skipped 3 previous similar messages [ 1236.047891] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 1248.634376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1248.714648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.847618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1248.851854] Lustre: Skipped 1 previous similar message [ 1248.914681] Lustre: lustre-MDT0001: new disk, initializing [ 1249.087916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1249.105830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1249.112515] Lustre: Skipped 1 previous similar message [ 1254.521996] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 1265.422866] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1272.034230] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1279.313578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1279.391431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1279.398928] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1279.633528] Lustre: lustre-OST0000: new disk, initializing [ 1279.639498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1281.289608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1281.304634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1281.353223] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1285.468156] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 1295.676572] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1305.091914] Lustre: lustre-OST0000-osc-MDT0001:0x280000400 is used up. Update to new seq [ 1305.098621] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000401 [ 1327.585771] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1327.586837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1327.598101] LustreError: Skipped 1 previous similar message [ 1327.601390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1327.615412] Lustre: Skipped 3 previous similar messages [ 1327.637684] Lustre: Skipped 3 previous similar messages [ 1336.288638] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1342.956216] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.975935] LustreError: Skipped 17 previous similar messages [ 1343.836083] LustreError: 45238:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193579 with bad export cookie 12600466244661817331 [ 1343.859047] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1358.951950] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1363.597363] Key type lgssc unregistered [ 1363.913137] LNet: 49321:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.929209] LNet: Removed LNI 192.168.202.145@tcp [ 1365.212458] Key type .llcrypt unregistered [ 1365.216369] Key type ._llcrypt unregistered [ 1397.941908] alg: No test for adler32 (adler32-zlib) [ 1398.694589] Key type ._llcrypt registered [ 1398.696526] Key type .llcrypt registered [ 1398.804106] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 1414.061477] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1415.615675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1416.163607] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 1416.453845] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1416.458096] LNet: Accept secure, port 988 [ 1418.183524] Key type lgssc registered [ 1419.140886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1430.383454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1438.245830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1446.833139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1454.756403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1466.105815] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1478.698515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1478.817542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.208221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1480.273849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1480.410293] Lustre: lustre-MDT0000: new disk, initializing [ 1480.500578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.532490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1485.983620] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 1498.666529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1498.751406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.985092] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1499.098182] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1499.107194] Lustre: Skipped 1 previous similar message [ 1499.265495] Lustre: lustre-MDT0001: new disk, initializing [ 1499.376141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1499.427099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1499.445546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1504.484795] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 1515.459900] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1522.987412] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1531.787430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1531.874276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1531.883811] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1532.231394] Lustre: lustre-OST0000: new disk, initializing [ 1532.243367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1532.324854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.693768] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 1542.247744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1542.259049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1542.337116] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1548.715215] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1555.168760] Lustre: DEBUG MARKER: On OST0, 49667 inodes available. Want 100500. rc=0 [ 1558.543079] Lustre: 55921:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1558.556718] Lustre: 55921:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1558.562392] Lustre: 55921:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1558.569869] Lustre: 55921:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1558.575682] Lustre: 55921:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1558.583537] Lustre: 55921:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1559.305157] Lustre: 55921:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 544, rollback = 9 [ 1559.317541] Lustre: 55921:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 63 previous similar messages [ 1559.327724] Lustre: 55921:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1559.334484] Lustre: 55921:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 63 previous similar messages [ 1559.341733] Lustre: 55921:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1559.347495] Lustre: 55921:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 63 previous similar messages [ 1559.365525] Lustre: 55921:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 1559.374887] Lustre: 55921:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 63 previous similar messages [ 1559.382155] Lustre: 55921:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1559.392931] Lustre: 55921:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 63 previous similar messages [ 1559.404980] Lustre: 55921:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1559.410517] Lustre: 55921:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 63 previous similar messages [ 1560.583703] Lustre: 55425:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 544, rollback = 9 [ 1560.596696] Lustre: 55425:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 95 previous similar messages [ 1560.604720] Lustre: 55425:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1560.608289] Lustre: 55425:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 1560.616896] Lustre: 55425:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1560.626094] Lustre: 55425:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 1560.633452] Lustre: 55425:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 1560.638269] Lustre: 55425:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 1560.647010] Lustre: 55425:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1560.657734] Lustre: 55425:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 1560.675945] Lustre: 55425:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1560.685437] Lustre: 55425:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 1562.668134] Lustre: 55921:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 544, rollback = 9 [ 1562.687198] Lustre: 55921:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 160 previous similar messages [ 1562.697287] Lustre: 55921:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1562.705290] Lustre: 55921:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 160 previous similar messages [ 1562.718635] Lustre: 55921:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1562.728539] Lustre: 55921:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 160 previous similar messages [ 1562.742256] Lustre: 55921:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 1562.754809] Lustre: 55921:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 160 previous similar messages [ 1562.765704] Lustre: 55921:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1562.780360] Lustre: 55921:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 160 previous similar messages [ 1562.788577] Lustre: 55921:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1562.796611] Lustre: 55921:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 160 previous similar messages [ 1567.299321] Lustre: 55424:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 1088, rollback = 9 [ 1567.318035] Lustre: 55424:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 351 previous similar messages [ 1567.332512] Lustre: 55424:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 64/256/0, destroy: 0/0/0 [ 1567.343124] Lustre: 55424:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 351 previous similar messages [ 1567.358183] Lustre: 55424:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1567.365672] Lustre: 55424:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 351 previous similar messages [ 1567.387982] Lustre: 55424:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 1567.402909] Lustre: 55424:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 351 previous similar messages [ 1567.412875] Lustre: 55424:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 64/1088/0, delete: 0/0/0 [ 1567.422053] Lustre: 55424:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 351 previous similar messages [ 1567.432342] Lustre: 55424:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1567.436808] Lustre: 55424:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 351 previous similar messages [ 1575.610232] Lustre: 55921:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 1088, rollback = 9 [ 1575.627865] Lustre: 55921:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 575 previous similar messages [ 1575.637801] Lustre: 55921:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 64/256/0, destroy: 0/0/0 [ 1575.644673] Lustre: 55921:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 575 previous similar messages [ 1575.653783] Lustre: 55921:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1575.657756] Lustre: 55921:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 575 previous similar messages [ 1575.675520] Lustre: 55921:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 1575.683304] Lustre: 55921:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 575 previous similar messages [ 1575.687912] Lustre: 55921:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 64/1088/0, delete: 0/0/0 [ 1575.694159] Lustre: 55921:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 575 previous similar messages [ 1575.708942] Lustre: 55921:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1575.721787] Lustre: 55921:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 575 previous similar messages [ 1592.067858] Lustre: 55422:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 1088, rollback = 9 [ 1592.084519] Lustre: 55422:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 1215 previous similar messages [ 1592.101970] Lustre: 55422:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 64/256/0, destroy: 0/0/0 [ 1592.111136] Lustre: 55422:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 1215 previous similar messages [ 1592.124504] Lustre: 55422:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1592.133368] Lustre: 55422:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 1215 previous similar messages [ 1592.145385] Lustre: 55422:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 1592.155394] Lustre: 55422:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 1215 previous similar messages [ 1592.167484] Lustre: 55422:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 64/1088/0, delete: 0/0/0 [ 1592.177698] Lustre: 55422:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 1215 previous similar messages [ 1592.183393] Lustre: 55422:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1592.187898] Lustre: 55422:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 1215 previous similar messages [ 1624.621198] Lustre: 55424:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 1088, rollback = 9 [ 1624.631632] Lustre: 55424:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 2239 previous similar messages [ 1624.638661] Lustre: 55424:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 64/256/0, destroy: 0/0/0 [ 1624.649947] Lustre: 55424:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 2239 previous similar messages [ 1624.660926] Lustre: 55424:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1624.667041] Lustre: 55424:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 2239 previous similar messages [ 1624.687396] Lustre: 55424:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 1624.697823] Lustre: 55424:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 2239 previous similar messages [ 1624.704762] Lustre: 55424:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 64/1088/0, delete: 0/0/0 [ 1624.717594] Lustre: 55424:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 2239 previous similar messages [ 1624.725426] Lustre: 55424:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1624.734842] Lustre: 55424:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 2239 previous similar messages [ 1688.896568] Lustre: 55425:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 1088, rollback = 9 [ 1688.919989] Lustre: 55425:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 4863 previous similar messages [ 1688.929599] Lustre: 55425:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 64/256/0, destroy: 0/0/0 [ 1688.937827] Lustre: 55425:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 4863 previous similar messages [ 1688.946417] Lustre: 55425:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1688.951900] Lustre: 55425:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 4863 previous similar messages [ 1688.964362] Lustre: 55425:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 1688.974295] Lustre: 55425:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 4863 previous similar messages [ 1688.977115] Lustre: 55425:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 64/1088/0, delete: 0/0/0 [ 1688.985810] Lustre: 55425:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 4863 previous similar messages [ 1688.993582] Lustre: 55425:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1689.006623] Lustre: 55425:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 4863 previous similar messages [ 1817.413358] Lustre: 55422:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 1088, rollback = 9 [ 1817.428766] Lustre: 55422:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 2495 previous similar messages [ 1817.443851] Lustre: 55422:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 64/256/0, destroy: 0/0/0 [ 1817.453415] Lustre: 55422:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 2495 previous similar messages [ 1817.465399] Lustre: 55422:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1817.471958] Lustre: 55422:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 2495 previous similar messages [ 1817.486420] Lustre: 55422:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 1817.509753] Lustre: 55422:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 2495 previous similar messages [ 1817.517302] Lustre: 55422:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 64/1088/0, delete: 0/0/0 [ 1817.526615] Lustre: 55422:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 2495 previous similar messages [ 1817.531604] Lustre: 55422:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1817.541409] Lustre: 55422:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 2495 previous similar messages [ 2073.485649] Lustre: 55424:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 1088, rollback = 9 [ 2073.499334] Lustre: 55424:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 10687 previous similar messages [ 2073.507238] Lustre: 55424:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 64/256/0, destroy: 0/0/0 [ 2073.511056] Lustre: 55424:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 10687 previous similar messages [ 2073.521571] Lustre: 55424:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 2073.525199] Lustre: 55424:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 10687 previous similar messages [ 2073.528127] Lustre: 55424:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 2073.535689] Lustre: 55424:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 10687 previous similar messages [ 2073.542895] Lustre: 55424:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 64/1088/0, delete: 0/0/0 [ 2073.552435] Lustre: 55424:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 10687 previous similar messages [ 2073.559162] Lustre: 55424:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2073.566723] Lustre: 55424:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 10687 previous similar messages [ 2586.125446] Lustre: 55422:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 1088, rollback = 9 [ 2586.136961] Lustre: 55422:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 24319 previous similar messages [ 2586.156732] Lustre: 55422:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 64/256/0, destroy: 0/0/0 [ 2586.170547] Lustre: 55422:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 24319 previous similar messages [ 2586.186467] Lustre: 55422:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 2586.198484] Lustre: 55422:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 24319 previous similar messages [ 2586.210164] Lustre: 55422:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 2586.222560] Lustre: 55422:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 24319 previous similar messages [ 2586.239552] Lustre: 55422:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 64/1088/0, delete: 0/0/0 [ 2586.255489] Lustre: 55422:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 24319 previous similar messages [ 2586.266480] Lustre: 55422:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2586.277721] Lustre: 55422:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 24319 previous similar messages [ 3194.169790] Lustre: 55424:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 1088, rollback = 9 [ 3194.187709] Lustre: 55424:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 22975 previous similar messages [ 3194.196282] Lustre: 55424:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 64/256/1, destroy: 0/0/0 [ 3194.205083] Lustre: 55424:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 22975 previous similar messages [ 3194.214371] Lustre: 55424:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 3194.219704] Lustre: 55424:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 22975 previous similar messages [ 3194.226708] Lustre: 55424:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 3194.234387] Lustre: 55424:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 22975 previous similar messages [ 3194.246089] Lustre: 55424:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 64/1088/0, delete: 0/0/0 [ 3194.254921] Lustre: 55424:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 22975 previous similar messages [ 3194.269555] Lustre: 55424:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3194.274166] Lustre: 55424:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 22975 previous similar messages [ 3794.542898] Lustre: 55422:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 1088, rollback = 9 [ 3794.548015] Lustre: 55422:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 29695 previous similar messages [ 3794.551559] Lustre: 55422:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 64/256/0, destroy: 0/0/0 [ 3794.559670] Lustre: 55422:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 29695 previous similar messages [ 3794.565390] Lustre: 55422:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 3794.577849] Lustre: 55422:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 29695 previous similar messages [ 3794.589076] Lustre: 55422:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 3794.594056] Lustre: 55422:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 29695 previous similar messages [ 3794.600567] Lustre: 55422:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 64/1088/0, delete: 0/0/0 [ 3794.607950] Lustre: 55422:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 29695 previous similar messages [ 3794.618592] Lustre: 55422:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3794.625153] Lustre: 55422:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 29695 previous similar messages [ 4132.509322] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4135.478977] Lustre: Failing over lustre-OST0000 [ 4135.564636] Lustre: server umount lustre-OST0000 complete [ 4144.611670] Lustre: 51038:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196373/real 1776196373] req@000000009f597405 x1862474041342848/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776196380 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4144.656558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.064873] LDISKFS-fs (dm-2): recovery complete [ 4162.068927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4162.398928] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4162.423880] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4167.003948] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4168.179648] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 4178.067942] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.145@tcp (at 0@lo) [ 4178.071200] Lustre: lustre-OST0000: Recovery over after 0:11, of 3 clients 3 recovered and 0 were evicted. [ 4178.074233] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 4178.085648] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4295077042 to 0x0:4295077121 [ 4182.175649] Lustre: 51039:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196373/real 1776196373] req@00000000c57166a5 x1862474041342656/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776196417 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4182.223943] Lustre: 51039:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4185.671873] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4187.898737] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4188.067385] Lustre: 51039:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196379/real 1776196379] req@00000000ff3af288 x1862474041343424/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776196423 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4203.492208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4203.497650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4203.524273] Lustre: Skipped 2 previous similar messages [ 4203.538035] Lustre: Skipped 1 previous similar message [ 4208.610496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4208.629950] Lustre: Skipped 1 previous similar message [ 4208.907646] Lustre: server umount lustre-OST0000 complete [ 4213.734564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4213.749962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4218.357156] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4218.361081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.405229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4219.897864] Lustre: server umount lustre-MDT0000 complete [ 4223.967177] Lustre: 55916:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196413/real 1776196413] req@000000003e2a99b5 x1862474041350912/t0(0) o5->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 432/432 e 0 to 1 dl 1776196456 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4223.986526] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4223.997220] Lustre: 55916:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4223.997505] LustreError: 55916:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -11 [ 4224.014791] LustreError: Skipped 1 previous similar message [ 4224.435187] LustreError: 55439:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196459 with bad export cookie 13400851743675329890 [ 4224.435741] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4224.444287] LustreError: 55439:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4224.887241] Lustre: server umount lustre-MDT0001 complete [ 4235.388409] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4237.855427] Key type lgssc unregistered [ 4238.079558] LNet: 63530:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4238.086371] LNet: Removed LNI 192.168.202.145@tcp [ 4238.703965] Key type .llcrypt unregistered [ 4238.708531] Key type ._llcrypt unregistered [ 4259.219811] alg: No test for adler32 (adler32-zlib) [ 4259.979812] Key type ._llcrypt registered [ 4259.983644] Key type .llcrypt registered [ 4260.119431] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4261.354224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4261.952048] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 4262.167304] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4262.170388] LNet: Accept secure, port 988 [ 4263.815165] Key type lgssc registered [ 4265.190474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4273.034367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.622521] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.714593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4279.787048] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4281.357864] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 4284.897873] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.023826] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4292.497454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.149480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4299.697592] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 4308.344559] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4316.541388] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4323.839430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4324.111377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.168314] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4295077222 to 0x0:4295077249 [ 4329.208091] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 4338.386126] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4355.620934] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:56:29 (1776196589) [ 4357.938453] Lustre: Modifying parameter general.jobid_name in log params [ 4359.767528] Lustre: Modifying parameter general.jobid_name in log params [ 4366.723505] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:56:39 (1776196599) [ 4376.660345] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:56:49 (1776196609) [ 4379.032160] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb in log lustre-client [ 4389.090469] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:57:02 (1776196622) [ 4392.272402] Lustre: Setting parameter general.osc.*.max_pages_per_rpc in log params [ 4392.280518] Lustre: Skipped 2 previous similar messages [ 4401.190173] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc in log lustre-client [ 4401.203678] Lustre: Skipped 1 previous similar message [ 4415.519212] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc in log lustre-client [ 4415.526078] Lustre: Skipped 5 previous similar messages [ 4418.032736] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:57:31 (1776196651) [ 4421.255853] Lustre: *** cfs_fail_loc=131b, val=0*** [ 4424.846879] Lustre: *** cfs_fail_loc=131b, val=0*** [ 4433.341354] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:57:46 (1776196666) [ 4435.194926] Lustre: Modifying parameter general.jobid_name in log params [ 4435.199770] Lustre: Skipped 3 previous similar messages [ 4448.001732] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:58:01 (1776196681) [ 4467.818931] Lustre: Modifying parameter general.timeout in log params [ 4467.821311] Lustre: Skipped 21 previous similar messages [ 4531.864596] Lustre: Modifying parameter general.timeout in log params [ 4531.874529] Lustre: Skipped 50 previous similar messages [ 4610.649403] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 16:00:44 (1776196844) [ 4625.889236] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4625.904882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.922271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4627.939084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.946742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4627.956406] Lustre: Skipped 1 previous similar message [ 4627.979855] Lustre: Skipped 2 previous similar messages [ 4629.626625] Lustre: server umount lustre-MDT0000 complete [ 4633.063773] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.076357] LustreError: Skipped 2 previous similar messages [ 4634.515932] LustreError: 64846:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196870 with bad export cookie 7849049574252983372 [ 4634.527706] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4634.536214] LustreError: 64846:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4635.223639] Lustre: server umount lustre-MDT0001 complete [ 4646.367756] Lustre: 79111:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196875/real 1776196875] req@00000000853385e9 x1862477018137024/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776196881 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'jobid_name=jobid_name=umount.ro' [ 4647.044479] Lustre: server umount lustre-OST0000 complete [ 4658.663862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4663.283809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4668.089705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4672.973541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4681.171904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4681.435273] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4681.458702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4681.469123] Lustre: Skipped 62 previous similar messages [ 4681.842851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4687.138490] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 4698.174728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.317291] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4698.954653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4705.178268] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 4714.570535] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4721.432712] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4729.687909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4730.155270] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4730.724771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4736.082249] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 4737.160583] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4295077222 to 0x0:4295077281 [ 4746.215928] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4757.472348] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4757.481771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4757.515497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4762.819942] Lustre: server umount lustre-OST0000 complete [ 4768.226823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4768.240969] Lustre: Skipped 1 previous similar message [ 4768.246471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4768.255940] Lustre: Skipped 2 previous similar messages [ 4773.344952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4773.352622] Lustre: Skipped 1 previous similar message [ 4774.065447] Lustre: server umount lustre-MDT0000 complete [ 4774.374494] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4774.399077] LustreError: Skipped 1 previous similar message [ 4778.013636] LustreError: 80551:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197013 with bad export cookie 7849049574252987012 [ 4778.016462] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4778.029800] LustreError: 80551:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4794.200777] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4797.193634] Key type lgssc unregistered [ 4797.577753] LNet: 84482:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4797.587199] LNet: Removed LNI 192.168.202.145@tcp [ 4798.665522] Key type .llcrypt unregistered [ 4798.676401] Key type ._llcrypt unregistered [ 4872.230356] alg: No test for adler32 (adler32-zlib) [ 4872.984543] Key type ._llcrypt registered [ 4872.987968] Key type .llcrypt registered [ 4873.095548] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4873.435991] Key type .llcrypt unregistered [ 4873.454801] Key type ._llcrypt unregistered [ 4878.549739] alg: No test for adler32 (adler32-zlib) [ 4879.303944] Key type ._llcrypt registered [ 4879.306180] Key type .llcrypt registered [ 4879.435710] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 4885.432936] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules [ 4887.433646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4888.148491] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 4889.079314] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4889.084988] LNet: Accept secure, port 988 [ 4891.021369] Key type lgssc registered [ 4892.305727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4894.111911] LustreError: 88873:0:(libcfs_fail.h:202:cfs_race_wait()) cfs_race id 60d sleeping [ 4910.617312] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4916.610884] LustreError: 88873:0:(libcfs_fail.h:205:cfs_race_wait()) cfs_fail_race id 60d awake: rc=0 [ 4921.801575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4923.479967] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4923.688290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4929.005678] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4929.182981] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 4934.114554] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4938.726881] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:06:12 (1776197172) [ 4939.237114] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4943.734565] LustreError: 89983:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4943.751081] LustreError: 89983:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4943.758953] LustreError: 89983:0:(lod_dev.c:528:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4949.983798] Lustre: 90825:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197179/real 1776197179] req@000000005ec9f9cf x1862477675499264/t0(0) o251->MGC192.168.202.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776197185 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4950.312098] Lustre: server umount lustre-MDT0000 complete [ 4964.820745] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4967.942728] Key type lgssc unregistered [ 4968.212359] LNet: 91581:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4968.233524] LNet: Removed LNI 192.168.202.145@tcp [ 4969.229273] Key type .llcrypt unregistered [ 4969.230731] Key type ._llcrypt unregistered [ 4986.831495] alg: No test for adler32 (adler32-zlib) [ 4987.597612] Key type ._llcrypt registered [ 4987.601197] Key type .llcrypt registered [ 4987.721074] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4988.857428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4989.883941] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 4990.240231] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4990.249541] LNet: Accept secure, port 988 [ 4991.919250] Key type lgssc registered [ 4993.230463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5002.440304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5004.199598] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5004.401174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5009.525520] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5009.901159] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5015.034787] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5019.740749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.432453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5026.851344] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5036.901583] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5044.611792] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5053.086332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5053.608508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5055.681802] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4295077222 to 0x0:4295077313 [ 5058.787412] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5068.019667] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5118.517380] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:09:11 (1776197351) [ 5120.578364] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 5122.726095] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:09:15 (1776197355) [ 5127.662177] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5127.673632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.694819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5131.745817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.746864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5131.754136] Lustre: Skipped 1 previous similar message [ 5131.758058] Lustre: Skipped 1 previous similar message [ 5133.465557] Lustre: server umount lustre-MDT0000 complete [ 5136.867277] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5136.883745] LustreError: Skipped 2 previous similar messages [ 5136.943799] LustreError: 92852:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197372 with bad export cookie 3485608379770604977 [ 5136.947787] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5136.959820] LustreError: 92852:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5137.347702] Lustre: server umount lustre-MDT0001 complete [ 5148.128347] Lustre: 96359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197377/real 1776197377] req@0000000096d394b7 x1862477781428480/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776197383 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5148.280947] Lustre: server umount lustre-OST0000 complete [ 5159.709365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.122309] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.291153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5164.359869] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5170.663788] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5170.691904] LustreError: Skipped 1 previous similar message [ 5173.932809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5174.883589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5181.040874] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5190.325689] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5197.580803] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5204.951868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5213.297678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5213.432981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5213.445297] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5213.963530] LustreError: 140-5: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 5213.978260] LustreError: 96977:0:(mgs_handler.c:503:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 5213.995566] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 5214.003821] LustreError: 99194:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -98 [ 5214.009716] LustreError: 99194:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 5214.014022] LustreError: 99194:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 5214.128553] Lustre: server umount lustre-OST0000 complete [ 5214.130912] LustreError: 99194:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 5224.178572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5224.363693] LustreError: 140-5: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 5224.373987] LustreError: 96975:0:(mgs_handler.c:503:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 5224.409945] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 5224.417452] LustreError: 99563:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -98 [ 5224.421575] LustreError: 99563:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 5224.428356] LustreError: 99563:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 5224.556338] Lustre: server umount lustre-OST0000 complete [ 5224.566119] LustreError: 99563:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 5226.892215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5237.475837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5237.963830] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5237.968453] Lustre: Found index 0 for lustre-OST0000, updating log [ 5237.977732] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 5238.165588] Lustre: lustre-OST0000: new disk, initializing [ 5238.170371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5238.696133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.638855] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5249.974353] Lustre: Failing over lustre-OST0000 [ 5250.284660] Lustre: server umount lustre-OST0000 complete [ 5255.138423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.140158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5255.151301] Lustre: Skipped 1 previous similar message [ 5255.154029] Lustre: Skipped 1 previous similar message [ 5260.256414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5260.266929] Lustre: Skipped 2 previous similar messages [ 5262.175161] Lustre: 92184:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197490/real 1776197490] req@00000000268a1b67 x1862477781452288/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776197497 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 5262.190680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5262.212706] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5262.225573] LustreError: Skipped 1 previous similar message [ 5265.343713] LustreError: 96967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197500 with bad export cookie 3485608379770607896 [ 5265.344240] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5265.354878] LustreError: 96967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5275.344427] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:11:48 (1776197508) [ 5293.797889] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5308.190424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5308.695149] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5308.943154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5313.824390] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5322.103236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5326.914586] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5335.895863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5336.526350] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5338.532963] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5338.550510] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5338.585044] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 5338.614613] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5338.621220] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 5342.304771] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5343.782415] Lustre: lustre-OST0000: precreate FID 0x0:4295077314 is over 4295077314 higher than LAST_ID 0x0:0, only precreating the last 10000 objects. OST replaced or reformatted? [ 5353.489803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5353.618677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5353.632603] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5353.864901] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5353.982470] Lustre: lustre-OST0001: new disk, initializing [ 5353.988823] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5354.261360] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5354.280196] Lustre: Skipped 2 previous similar messages [ 5360.168265] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5362.316891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5362.339583] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5371.010344] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5378.988541] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 5388.263043] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5388.275924] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5388.287163] Lustre: Skipped 1 previous similar message [ 5392.702867] Lustre: server umount lustre-MDT0001 complete [ 5392.705757] Lustre: Skipped 2 previous similar messages [ 5393.378709] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5393.406134] LustreError: Skipped 4 previous similar messages [ 5396.045052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5403.350934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.457637] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5403.461552] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5403.534276] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5403.617223] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5403.626846] Lustre: Skipped 5 previous similar messages [ 5405.193840] LustreError: 106157:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0001: already exists, won't add [ 5405.209657] LustreError: 106157:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 5405.221725] LustreError: 106157:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 5405.234534] LustreError: 102351:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5405.341218] LustreError: 106164:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.145@tcp: cfg command failed: rc = -17 [ 5405.349423] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5405.349423] [ 5409.140388] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5409.259124] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5409.275648] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5409.282078] Lustre: Skipped 1 previous similar message [ 5419.096578] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 16:14:12 (1776197652) [ 5436.039663] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5474.420629] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5477.933981] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5541.856271] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5541.865673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5541.881422] Lustre: Skipped 1 previous similar message [ 5541.891695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5545.230556] Lustre: server umount lustre-MDT0000 complete [ 5547.496027] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5547.520523] LustreError: Skipped 8 previous similar messages [ 5549.233360] LustreError: 104697:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197784 with bad export cookie 3485608379770608883 [ 5549.234342] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5549.248446] LustreError: 104697:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5559.775212] Lustre: 110013:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197789/real 1776197789] req@000000006a0b393d x1862477781567552/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776197795 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5559.775336] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.792744] Lustre: 110013:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5559.831645] Lustre: Skipped 3 previous similar messages [ 5561.889262] Lustre: 92181:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197790/real 1776197790] req@00000000393e9971 x1862477781567680/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776197797 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5571.144798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5580.425090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5581.365824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5590.029418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5598.994188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5600.203220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5624.520117] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5636.317394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.343790] Lustre: lustre-MDT0000: reset Object Index mappings [ 5637.071245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5637.077491] Lustre: Skipped 1 previous similar message [ 5642.537401] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5653.684454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.743249] Lustre: lustre-MDT0001: reset Object Index mappings [ 5660.233275] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5669.359643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5675.225595] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5684.228974] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:226 to 0x280000400:257 [ 5685.755839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5687.350426] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:194 to 0x2c0000400:225 [ 5687.352058] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4295077475 to 0x0:4295077506 [ 5687.593120] Lustre: lustre-OST0001: deleting orphan objects from 0x0:162 to 0x0:193 [ 5691.482760] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5701.282793] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5710.221137] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5767.002308] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:19:59 (1776197999) [ 5787.107763] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5787.117122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.129952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5787.134426] Lustre: Skipped 4 previous similar messages [ 5799.391178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5799.545313] Lustre: server umount lustre-MDT0000 complete [ 5799.557822] Lustre: Skipped 3 previous similar messages [ 5803.105477] LustreError: 113126:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198038 with bad export cookie 3485608379770664253 [ 5803.114398] LustreError: 166-1: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5805.030228] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.045532] LustreError: Skipped 11 previous similar messages [ 5838.305209] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5850.052286] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 5863.485665] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5874.268734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5882.032066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5889.440821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5896.054317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5903.735926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5903.807977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5904.089880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5904.119669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5904.219733] Lustre: lustre-MDT0000: new disk, initializing [ 5904.312896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5904.323517] Lustre: Skipped 3 previous similar messages [ 5904.344371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5908.552287] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5921.248037] Lustre: 120129:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198150/real 1776198150] req@00000000e99d1cbe x1862477781622848/t0(0) o251->MGC192.168.202.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776198156 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5924.279852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5930.497571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5930.568814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.797621] Lustre: Found index 0 for lustre-MDT0000, updating log [ 5930.813790] Lustre: Modifying parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5936.133995] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all 8 [ 5947.359259] Lustre: 121229:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198176/real 1776198176] req@00000000f6bcf509 x1862477781625152/t0(0) o251->MGC192.168.202.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776198182 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5947.540625] Lustre: server umount lustre-MDT0000 complete [ 5947.545023] Lustre: Skipped 4 previous similar messages [ 5975.516928] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 5984.917860] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5998.977167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6007.143781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6013.151375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6020.281074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6022.141658] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5764 sec ========== 16:24:15 (1776198255) [ 6061.472788] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 6064.293803] Key type lgssc unregistered [ 6064.623188] LNet: 126084:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6064.629578] LNet: Removed LNI 192.168.202.145@tcp [ 6065.327395] Key type .llcrypt unregistered [ 6065.329704] Key type ._llcrypt unregistered