[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 640584340 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003250] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005129] kvm-guest: setup PV IPIs [ 0.009474] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011032] pid_max: default: 32768 minimum: 301 [ 0.012135] LSM: Security Framework initializing [ 0.013067] Yama: becoming mindful. [ 0.014049] SELinux: Initializing. [ 0.015129] *** VALIDATE selinux *** [ 0.024732] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030116] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031213] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033229] *** VALIDATE tmpfs *** [ 0.034000] *** VALIDATE proc *** [ 0.034286] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.037115] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039038] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041000] Spectre V2 : User space: Vulnerable [ 0.041000] Speculative Store Bypass: Vulnerable [ 0.044049] debug: unmapping init [mem 0xffffffff85a59000-0xffffffff85a60fff] [ 0.046240] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049821] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050021] ... version: 2 [ 0.051019] ... bit width: 48 [ 0.052015] ... generic registers: 4 [ 0.053315] ... value mask: 0000ffffffffffff [ 0.054018] ... max period: 00007fffffffffff [ 0.055018] ... fixed-purpose events: 3 [ 0.056008] ... event mask: 000000070000000f [ 0.057273] rcu: Hierarchical SRCU implementation. [ 0.060256] smp: Bringing up secondary CPUs ... [ 0.062219] x86: Booting SMP configuration: [ 0.063024] .... node #0, CPUs: #1 #2 #3 [ 0.079781] smp: Brought up 1 node, 4 CPUs [ 0.081012] smpboot: Max logical packages: 1 [ 0.082020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.114141] node 0 deferred pages initialised in 28ms [ 0.121034] devtmpfs: initialized [ 0.122290] x86/mm: Memory block size: 128MB [ 0.125431] gcov: version magic: 0x41383552 [ 0.127507] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.128099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.129256] pinctrl core: initialized pinctrl subsystem [ 0.130238] [ 0.131007] ************************************************************* [ 0.132011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.133011] ** ** [ 0.134011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.135011] ** ** [ 0.136014] ** This means that this kernel is built to expose internal ** [ 0.137013] ** IOMMU data structures, which may compromise security on ** [ 0.138018] ** your system. ** [ 0.139016] ** ** [ 0.140014] ** If you see this message and you are not debugging the ** [ 0.141016] ** kernel, report this immediately to your vendor! ** [ 0.142012] ** ** [ 0.143014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144014] ************************************************************* [ 0.145945] NET: Registered protocol family 16 [ 0.146540] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.147070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.148065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.150705] cpuidle: using governor menu [ 0.152532] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.153909] PCI: Using configuration type 1 for base access [ 0.154189] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.166181] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.167000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.178042] cryptd: max_cpu_qlen set to 1000 [ 0.181282] ACPI: Added _OSI(Module Device) [ 0.182011] ACPI: Added _OSI(Processor Device) [ 0.183016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.184013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.188917] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.194245] ACPI: Interpreter enabled [ 0.195295] ACPI: PM: (supports S0 S3 S4 S5) [ 0.196012] ACPI: Using IOAPIC for interrupt routing [ 0.197109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.198256] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.209107] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.210165] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.211015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.212133] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.214705] acpiphp: Slot [2] registered [ 0.215154] acpiphp: Slot [5] registered [ 0.216083] acpiphp: Slot [6] registered [ 0.217084] acpiphp: Slot [7] registered [ 0.218092] acpiphp: Slot [8] registered [ 0.219096] acpiphp: Slot [9] registered [ 0.220132] acpiphp: Slot [10] registered [ 0.221153] acpiphp: Slot [3] registered [ 0.222144] acpiphp: Slot [4] registered [ 0.223109] acpiphp: Slot [11] registered [ 0.224096] acpiphp: Slot [12] registered [ 0.225320] acpiphp: Slot [13] registered [ 0.226102] acpiphp: Slot [14] registered [ 0.227087] acpiphp: Slot [15] registered [ 0.228161] acpiphp: Slot [16] registered [ 0.229109] acpiphp: Slot [17] registered [ 0.230080] acpiphp: Slot [18] registered [ 0.231083] acpiphp: Slot [19] registered [ 0.232100] acpiphp: Slot [20] registered [ 0.233237] acpiphp: Slot [21] registered [ 0.234251] acpiphp: Slot [22] registered [ 0.235260] acpiphp: Slot [23] registered [ 0.236093] acpiphp: Slot [24] registered [ 0.237083] acpiphp: Slot [25] registered [ 0.238171] acpiphp: Slot [26] registered [ 0.239103] acpiphp: Slot [27] registered [ 0.240082] acpiphp: Slot [28] registered [ 0.241106] acpiphp: Slot [29] registered [ 0.242118] acpiphp: Slot [30] registered [ 0.243115] acpiphp: Slot [31] registered [ 0.244076] PCI host bridge to bus 0000:00 [ 0.245019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.246019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.247018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.248019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.249024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.250022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.251156] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.253493] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.255519] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.261013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.264718] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.265012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.266016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.267014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.269082] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.270770] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.271097] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.273044] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.276013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.284045] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.287047] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.290867] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.300019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.317016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.341000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.353267] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.361016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.370021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.396015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.413000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.429015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.443015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.477015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.490239] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.507016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.516016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.543016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.566000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.579014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.585015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.604000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.611853] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.623014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.636012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.673023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.693637] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.696506] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.702629] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.706970] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.712434] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.721000] iommu: Default domain type: Passthrough [ 0.722864] SCSI subsystem initialized [ 0.724149] ACPI: bus type USB registered [ 0.727127] usbcore: registered new interface driver usbfs [ 0.729076] usbcore: registered new interface driver hub [ 0.731236] usbcore: registered new device driver usb [ 0.737109] pps_core: LinuxPPS API ver. 1 registered [ 0.740010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.747056] PTP clock support registered [ 0.752124] EDAC MC: Ver: 3.0.0 [ 0.755095] PCI: Using ACPI for IRQ routing [ 0.756721] NetLabel: Initializing [ 0.759011] NetLabel: domain hash size = 128 [ 0.760000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.761081] NetLabel: unlabeled traffic allowed by default [ 0.764106] vgaarb: loaded [ 0.767807] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.770011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.779290] clocksource: Switched to clocksource kvm-clock [ 0.967307] VFS: Disk quotas dquot_6.6.0 [ 0.968782] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.972305] *** VALIDATE ramfs *** [ 0.974144] *** VALIDATE hugetlbfs *** [ 0.976858] pnp: PnP ACPI init [ 0.980674] pnp: PnP ACPI: found 6 devices [ 1.004518] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.016815] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.020766] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.025087] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.027642] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.030253] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.033694] NET: Registered protocol family 2 [ 1.040742] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.051995] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.062298] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.075434] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.085299] TCP: Hash tables configured (established 65536 bind 65536) [ 1.092172] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.102065] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.110416] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.123420] NET: Registered protocol family 1 [ 1.129448] RPC: Registered named UNIX socket transport module. [ 1.135130] RPC: Registered udp transport module. [ 1.143230] RPC: Registered tcp transport module. [ 1.146268] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.155662] NET: Registered protocol family 44 [ 1.159644] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.164255] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.173837] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.177813] PCI: CLS 0 bytes, default 64 [ 1.181191] Unpacking initramfs... [ 5.460044] debug: unmapping init [mem 0xffff8d087cc54000-0xffff8d087ffbffff] [ 5.475099] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.481473] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.494378] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 8.263256] Initialise system trusted keyrings [ 8.265351] Key type blacklist registered [ 8.267609] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 8.279417] zbud: loaded [ 8.284465] *** VALIDATE nfs *** [ 8.285645] *** VALIDATE nfs4 *** [ 8.287482] pstore: using deflate compression [ 8.293176] Platform Keyring initialized [ 8.820132] NET: Registered protocol family 38 [ 8.823629] Key type asymmetric registered [ 8.825645] Asymmetric key parser 'x509' registered [ 8.828246] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 8.833191] io scheduler mq-deadline registered [ 8.836589] io scheduler kyber registered [ 8.845803] io scheduler bfq registered [ 8.851265] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 8.858316] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 8.869316] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 8.878692] ACPI: Power Button [PWRF] [ 8.890447] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.921220] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 9.020293] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 9.047468] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 9.107751] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 9.202914] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 9.300374] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 9.333273] Non-volatile memory driver v1.3 [ 9.342128] Linux agpgart interface v0.103 [ 9.452231] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.8 MB/73.2 MiB) [ 9.467506] vda: detected capacity change from 0 to 76787712 [ 9.552038] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 9.561837] vdb: detected capacity change from 0 to 1073741824 [ 9.621280] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.634032] vdc: detected capacity change from 0 to 2621440000 [ 9.689577] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.698633] vdd: detected capacity change from 0 to 2621440000 [ 9.753956] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.771816] vde: detected capacity change from 0 to 4294967296 [ 9.832472] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.841209] vdf: detected capacity change from 0 to 4294967296 [ 9.878856] libphy: Fixed MDIO Bus: probed [ 9.921523] usbcore: registered new interface driver usbserial_generic [ 9.940079] usbserial: USB Serial support registered for generic [ 9.946394] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.967517] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.979043] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.988368] mousedev: PS/2 mouse device common for all mice [ 9.999599] rtc_cmos 00:05: RTC can wake from S4 [ 10.002486] rtc_cmos 00:05: registered as rtc0 [ 10.009653] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 10.015606] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 10.027394] intel_pstate: CPU model not supported [ 10.042329] hid: raw HID events driver (C) Jiri Kosina [ 10.044345] usbcore: registered new interface driver usbhid [ 10.053610] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 10.056449] usbhid: USB HID core driver [ 10.056760] drop_monitor: Initializing network drop monitor service [ 10.056876] Initializing XFRM netlink socket [ 10.057284] NET: Registered protocol family 10 [ 10.058504] Segment Routing with IPv6 [ 10.058537] NET: Registered protocol family 17 [ 10.058839] mpls_gso: MPLS GSO support [ 10.137152] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 10.166805] RAS: Correctable Errors collector initialized. [ 10.183114] AVX version of gcm_enc/dec engaged. [ 10.185062] AES CTR mode by8 optimization enabled [ 10.503877] sched_clock: Marking stable (10503853730, 0)->(12172251362, -1668397632) [ 10.538478] registered taskstats version 1 [ 10.560436] Loading compiled-in X.509 certificates [ 10.567188] zswap: loaded using pool lzo/zbud [ 10.728212] Key type big_key registered [ 10.810425] Key type encrypted registered [ 10.816288] ima: No TPM chip found, activating TPM-bypass! [ 10.827625] ima: Allocated hash algorithm: sha1 [ 10.832651] ima: No architecture policies found [ 10.839796] evm: Initialising EVM extended attributes: [ 10.845479] evm: security.selinux [ 10.853436] evm: security.ima [ 10.859665] evm: security.capability [ 10.865136] evm: HMAC attrs: 0x1 [ 10.872768] rtc_cmos 00:05: setting system clock to 2026-09-17 22:11:21 UTC (1789683081) [ 10.903995] debug: unmapping init [mem 0xffffffff86a03000-0xffffffff86bfffff] [ 10.941434] debug: unmapping init [mem 0xffffffff85782000-0xffffffff85a58fff] [ 10.964554] Write protecting the kernel read-only data: 28672k [ 10.994735] debug: unmapping init [mem 0xffffffff83e03000-0xffffffff83ffffff] [ 11.004090] debug: unmapping init [mem 0xffffffff84714000-0xffffffff847fffff] [ 11.121297] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.149554] systemd[1]: Detected virtualization kvm. [ 11.156798] systemd[1]: Detected architecture x86-64. [ 11.162362] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 11.237471] systemd[1]: No hostname configured. [ 11.243341] systemd[1]: Set hostname to . [ 11.249042] random: systemd: uninitialized urandom read (16 bytes read) [ 11.256550] systemd[1]: Initializing machine ID from random generator. [ 11.488895] random: ln: uninitialized urandom read (6 bytes read) [ 11.819971] random: systemd: uninitialized urandom read (16 bytes read) [ 11.828649] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 11.853965] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 11.868436] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 14.269413] device-mapper: uevent: version 1.0.3 [ 14.280173] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Mounted Kernel Configuration File System. [ 15.620870] virtio_net virtio0 ens2: renamed from eth0 [ 15.751692] random: fast init done [ 15.958827] scsi host0: ata_piix [ 15.990251] scsi host1: ata_piix [ 15.999258] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 16.009253] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.693091] dracut-initqueue[485]: RTNETLINK answers: File exists [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 22.398904] random: crng init done [ 22.400476] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 25.327762] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 27.966758] printk: systemd: 26 output lines suppressed due to ratelimiting [ 29.182209] SELinux: Disabled at runtime. [ 29.302649] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 29.314186] systemd[1]: Detected virtualization kvm. [ 29.318156] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 31.434710] systemd[1]: initrd-switch-root.service: Succeeded. [ 31.443141] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 31.475442] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 31.491657] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 31.501422] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 31.516868] systemd[1]: Starting Journal Service... Starting Journal Service... [ 31.550630] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ 31.827365] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ 32.211801] hrtimer: interrupt took 3999264 ns [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 33.134306] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 34.584278] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 34.652624] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 35.470520] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.521623] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 40.464522] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 41.226212] NFS: Registering the id_resolver key type [ 41.228897] Key type id_resolver registered [ 41.230895] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg346-server login: [ 82.916213] libcfs: loading out-of-tree module taints kernel. [ 82.942666] Key type ._llcrypt registered [ 82.943860] Key type .llcrypt registered [ 83.000797] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 95.979589] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 97.088381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 97.115022] alg: No test for adler32 (adler32-zlib) [ 98.270527] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 98.866893] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 100.567385] Key type lgssc registered [ 101.729045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.766885] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.819063] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 151.827538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 151.852779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.059776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 153.088544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 153.167328] Lustre: lustre-MDT0000: new disk, initializing [ 153.222106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.235086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 156.099320] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.692485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.746974] Lustre: 6498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 165.779795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 165.783624] Lustre: Skipped 1 previous similar message [ 165.787084] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 165.837783] Lustre: lustre-MDT0001: new disk, initializing [ 165.874501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 165.896618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 165.904584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.451358] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.893430] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 184.856475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.228469] Lustre: lustre-OST0000: new disk, initializing [ 185.249065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.262325] Lustre: 8436:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 185.357407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.070168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.096933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.233303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 193.883992] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.698883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.172353] Lustre: lustre-OST0001: new disk, initializing [ 222.199732] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 222.215253] Lustre: 9511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.430944] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.105650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 232.123722] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.272621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.356286] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.645275] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 258.201434] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 265.337918] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.697752] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing yml_node [ 276.482527] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 279.713024] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 282.735086] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 284.494931] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Thu Sep 17 18:15:53 EDT 2026 [ 302.626878] Lustre: DEBUG MARKER: excepting tests: 18b 23b [ 312.569662] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 324.576600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.577707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 324.578073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.609936] Lustre: Skipped 2 previous similar messages [ 328.462817] Lustre: server umount lustre-MDT0000 complete [ 334.816184] LustreError: 7813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 334.828853] LustreError: 7813:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 336.559694] LustreError: 6491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683407 with bad export cookie 6775229265843560581 [ 336.565442] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 336.577281] LustreError: 6491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 337.036695] Lustre: server umount lustre-MDT0001 complete [ 356.127267] Lustre: 3635:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789683410/real 1789683410] req@ffff8d07c7d6e300 x1876618818725120/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789683426 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 356.169284] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.178086] Lustre: Skipped 1 previous similar message [ 357.362415] Lustre: server umount lustre-OST0000 complete [ 358.307855] Lustre: 3634:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789683412/real 1789683412] req@ffff8d07c7d6f480 x1876618818725376/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789683428 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 361.375233] Lustre: 3636:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789683415/real 1789683415] req@ffff8d07c7d6fb80 x1876618818725632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789683431 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 365.910416] Lustre: server umount lustre-OST0001 complete [ 383.150881] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 386.683965] Key type lgssc unregistered [ 387.044083] LNet: 14785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.053163] LNetError: 14785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 387.094265] LNet: Removed LNI 192.168.203.146@tcp [ 388.237391] Key type .llcrypt unregistered [ 388.239587] Key type ._llcrypt unregistered [ 413.015771] Key type ._llcrypt registered [ 413.017786] Key type .llcrypt registered [ 413.138661] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 428.895348] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 430.275148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 430.497995] alg: No test for adler32 (adler32-zlib) [ 431.762571] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 432.002877] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 433.751207] Key type lgssc registered [ 435.046976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 485.877456] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 499.413629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 499.490381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.757435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 500.797611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 500.901912] Lustre: lustre-MDT0000: new disk, initializing [ 501.001233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.037805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 506.483773] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 524.446469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.600045] Lustre: 19237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 524.628641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 524.632869] Lustre: Skipped 1 previous similar message [ 524.760513] Lustre: lustre-MDT0001: new disk, initializing [ 524.877450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 524.948746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 524.962676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 530.962836] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 536.373212] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 547.108200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 547.444511] Lustre: lustre-OST0000: new disk, initializing [ 547.450318] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 547.457234] Lustre: 21173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 547.562820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.950710] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 557.654780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 557.667055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 557.835017] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 573.282923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 573.440644] Lustre: lustre-OST0001: new disk, initializing [ 573.444378] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 573.450281] Lustre: 22197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 573.519460] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 580.203120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 580.210398] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 580.219872] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 580.320081] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 592.934615] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 601.036740] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 610.235594] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 18:21:18 (1789683678) === [ 613.838246] Lustre: DEBUG MARKER: == sanity-lfsck test 0: Control LFSCK manually =========== 18:21:22 (1789683682) [ 614.204588] Lustre: 19243:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 614.233377] Lustre: 19243:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 614.256371] Lustre: 19243:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 614.265593] Lustre: 19243:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 614.280523] Lustre: 19243:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 614.293227] Lustre: 19243:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 614.779189] Lustre: 19243:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 614.786516] Lustre: 19243:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 11 previous similar messages [ 614.793116] Lustre: 19243:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 614.801035] Lustre: 19243:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 614.806539] Lustre: 19243:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 614.812099] Lustre: 19243:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 614.817919] Lustre: 19243:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 614.824850] Lustre: 19243:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 614.831331] Lustre: 19243:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 614.838124] Lustre: 19243:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 614.848416] Lustre: 19243:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 614.854982] Lustre: 19243:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 615.792737] Lustre: 19243:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 615.804147] Lustre: 19243:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 56 previous similar messages [ 615.810751] Lustre: 19243:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 615.818919] Lustre: 19243:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 56 previous similar messages [ 615.823222] Lustre: 19243:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 615.830135] Lustre: 19243:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 56 previous similar messages [ 615.836298] Lustre: 19243:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 615.841820] Lustre: 19243:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 56 previous similar messages [ 615.847956] Lustre: 19243:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 615.852462] Lustre: 19243:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 56 previous similar messages [ 615.856220] Lustre: 19243:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 615.860237] Lustre: 19243:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 56 previous similar messages [ 617.805368] Lustre: 19243:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 617.814204] Lustre: 19243:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 95 previous similar messages [ 617.821693] Lustre: 19243:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 617.829582] Lustre: 19243:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 617.834278] Lustre: 19243:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 617.838794] Lustre: 19243:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 617.843772] Lustre: 19243:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 617.850477] Lustre: 19243:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 617.856191] Lustre: 19243:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 617.864218] Lustre: 19243:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 617.876757] Lustre: 19243:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 617.882087] Lustre: 19243:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 622.771903] Lustre: *** cfs_fail_loc=1600, val=3*** [ 624.839687] Lustre: 21164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 2: before 260 < left 274, rollback = 2 [ 624.844466] Lustre: 23386:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 624.855080] Lustre: 21164:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 113 previous similar messages [ 624.855108] Lustre: 21164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/274/0 [ 624.855113] Lustre: 21164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 112 previous similar messages [ 624.855118] Lustre: 21164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 624.855122] Lustre: 21164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 112 previous similar messages [ 624.855128] Lustre: 21164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 624.855132] Lustre: 21164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 112 previous similar messages [ 624.855138] Lustre: 21164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 624.855142] Lustre: 21164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 112 previous similar messages [ 625.036118] Lustre: 23386:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 121 previous similar messages [ 625.823129] Lustre: *** cfs_fail_loc=1600, val=3*** [ 628.805550] Lustre: *** cfs_fail_loc=1600, val=3*** [ 640.582160] Lustre: 23386:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 274, rollback = 2 [ 640.595949] Lustre: 23554:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 640.619464] Lustre: 23386:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 59 previous similar messages [ 640.625852] Lustre: 23554:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 50 previous similar messages [ 640.625875] Lustre: 23554:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 640.625879] Lustre: 23554:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 59 previous similar messages [ 640.625886] Lustre: 23554:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 640.625889] Lustre: 23554:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 59 previous similar messages [ 640.625895] Lustre: 23554:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 640.625898] Lustre: 23554:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 59 previous similar messages [ 640.625903] Lustre: 23554:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 640.625907] Lustre: 23554:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 59 previous similar messages [ 647.137792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.145395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.152047] Lustre: Skipped 1 previous similar message [ 647.171216] Lustre: Skipped 3 previous similar messages [ 652.256633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 652.271972] Lustre: Skipped 3 previous similar messages [ 657.375243] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 657.375934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 657.398983] Lustre: Skipped 3 previous similar messages [ 657.785262] Lustre: server umount lustre-MDT0000 complete [ 662.238870] LustreError: 19227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683732 with bad export cookie 732587196721057051 [ 662.256856] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 662.518912] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 662.549625] Lustre: Skipped 3 previous similar messages [ 662.657533] Lustre: server umount lustre-MDT0001 complete [ 669.400927] Lustre: server umount lustre-OST0000 complete [ 673.791431] Lustre: server umount lustre-OST0001 complete [ 685.208549] Lustre: DEBUG MARKER: == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 18:22:33 (1789683753) [ 702.241642] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 714.533181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.101798] LustreError: 26207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 715.168837] LustreError: 26207:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 715.286110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.935071] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 720.360094] LustreError: 26208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 724.448781] LustreError: 26207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 729.581525] LustreError: 26208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 729.976634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.372371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 735.448136] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 738.765266] Lustre: 27349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 745.649733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 752.194673] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 755.250154] LustreError: 27704:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 760.401316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 760.622729] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 760.626705] Lustre: Skipped 1 previous similar message [ 761.664611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:40 to 0x280000401:65) [ 761.675140] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:65) [ 766.440525] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 773.859336] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 777.386190] Lustre: 29219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 779.071759] Lustre: 26202:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 779.081027] Lustre: 26202:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 779.086259] Lustre: 26202:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 779.099123] Lustre: 26202:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 779.109866] Lustre: 26202:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 779.117225] Lustre: 26202:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 779.126103] Lustre: 26202:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 779.133914] Lustre: 26202:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 779.145429] Lustre: 26202:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 779.152948] Lustre: 26202:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 779.164816] Lustre: 26202:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 779.178576] Lustre: 26202:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 784.229455] Lustre: *** cfs_fail_loc=1501, val=0*** [ 791.892195] Lustre: Failing over lustre-MDT0000 [ 792.031981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 792.040953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.072689] LustreError: 26207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 792.092811] LustreError: 26207:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 792.183543] Lustre: server umount lustre-MDT0000 complete [ 802.831548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 803.010804] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 803.485275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 803.557193] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 808.932087] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 808.942670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 808.979757] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 809.056260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 809.058496] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 809.685118] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 813.107794] Lustre: *** cfs_fail_loc=1505, val=0*** [ 823.094882] Lustre: DEBUG MARKER: == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 18:24:51 (1789683891) [ 824.381222] Lustre: 27725:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 824.399894] Lustre: 27725:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 322 previous similar messages [ 824.405841] Lustre: 27725:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 824.422700] Lustre: 27725:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 824.435496] Lustre: 27725:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 824.441386] Lustre: 27725:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 824.447722] Lustre: 27725:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 824.453047] Lustre: 27725:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 824.456657] Lustre: 27725:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 824.469715] Lustre: 27725:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 824.482766] Lustre: 27725:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 824.494462] Lustre: 27725:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 831.427666] Lustre: *** cfs_fail_loc=1502, val=0*** [ 840.440029] Lustre: Failing over lustre-MDT0000 [ 840.753257] Lustre: server umount lustre-MDT0000 complete [ 844.770799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.772803] LustreError: 26943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 844.785670] Lustre: Skipped 6 previous similar messages [ 844.813373] LustreError: 26943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 853.362184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 853.618699] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 854.067821] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 859.135183] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 859.137175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 859.178521] Lustre: Skipped 3 previous similar messages [ 859.234666] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 859.319434] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:169 to 0x2c0000401:193) [ 859.325857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 859.970829] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 864.999352] Lustre: *** cfs_fail_loc=1505, val=0*** [ 874.632460] Lustre: DEBUG MARKER: == sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 18:25:42 (1789683942) [ 881.917154] Lustre: *** cfs_fail_loc=1504, val=0*** [ 881.921220] Lustre: *** cfs_fail_loc=1504, val=0*** [ 881.925969] Lustre: Skipped 1 previous similar message [ 889.856530] Lustre: Failing over lustre-MDT0000 [ 890.183165] Lustre: server umount lustre-MDT0000 complete [ 890.337785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 890.360126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.380637] LustreError: 26943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 890.398906] LustreError: 26943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 900.057026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 900.269793] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 900.652349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.656980] Lustre: Skipped 1 previous similar message [ 900.689729] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 905.421729] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 905.701721] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 905.708147] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 905.717185] Lustre: Skipped 3 previous similar messages [ 905.743670] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 905.772834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:233 to 0x280000401:257) [ 905.775127] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:257) [ 908.589626] Lustre: *** cfs_fail_loc=1505, val=0*** [ 916.476250] Lustre: DEBUG MARKER: == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 18:26:25 (1789683985) [ 917.705870] Lustre: 26203:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 917.710335] Lustre: 26203:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 644 previous similar messages [ 917.715370] Lustre: 26203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 917.719430] Lustre: 26203:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 917.723778] Lustre: 26203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 917.728519] Lustre: 26203:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 917.733970] Lustre: 26203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 917.740401] Lustre: 26203:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 917.751576] Lustre: 26203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 917.755984] Lustre: 26203:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 917.759790] Lustre: 26203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 917.774939] Lustre: 26203:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 923.099989] Lustre: *** cfs_fail_loc=1603, val=0*** [ 930.932517] Lustre: Failing over lustre-MDT0000 [ 931.189210] Lustre: server umount lustre-MDT0000 complete [ 931.296774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.303726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 931.311920] Lustre: Skipped 5 previous similar messages [ 942.498797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 942.735972] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 943.286433] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 948.713889] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 948.723495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 948.748238] Lustre: Skipped 3 previous similar messages [ 948.772805] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 948.835593] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:297 to 0x2c0000401:321) [ 948.836467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:297 to 0x280000401:321) [ 949.633721] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 963.906495] Lustre: DEBUG MARKER: == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 18:27:11 (1789684031) [ 971.010384] Lustre: *** cfs_fail_loc=1604, val=0*** [ 979.482552] Lustre: Failing over lustre-MDT0000 [ 980.015845] Lustre: server umount lustre-MDT0000 complete [ 984.547454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 984.555214] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.564941] LustreError: 26203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 984.564951] LustreError: 26203:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [ 984.660980] Lustre: Skipped 3 previous similar messages [ 993.025110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 993.256672] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 993.661504] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 998.882728] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 998.891655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 998.900588] Lustre: Skipped 3 previous similar messages [ 998.918548] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 998.969472] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:361 to 0x2c0000401:385) [ 998.969973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:361 to 0x280000401:385) [ 999.566422] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1011.120750] Lustre: DEBUG MARKER: == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 18:27:59 (1789684079) [ 1020.851938] Lustre: *** cfs_fail_loc=1605, val=0*** [ 1030.706429] Lustre: Failing over lustre-MDT0000 [ 1030.915462] Lustre: server umount lustre-MDT0000 complete [ 1034.726534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1034.748893] Lustre: Skipped 3 previous similar messages [ 1042.213608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1042.727743] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1043.350255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.359505] Lustre: Skipped 2 previous similar messages [ 1043.433701] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1048.550718] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1048.563090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1048.596455] Lustre: Skipped 3 previous similar messages [ 1048.667720] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1048.809437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:425 to 0x280000401:449) [ 1048.819094] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:425 to 0x2c0000401:449) [ 1050.354785] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1061.484447] Lustre: DEBUG MARKER: == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 18:28:50 (1789684130) [ 1062.949940] Lustre: 26202:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1062.962653] Lustre: 26202:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 968 previous similar messages [ 1062.970640] Lustre: 26202:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1062.981621] Lustre: 26202:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1062.994368] Lustre: 26202:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1063.004422] Lustre: 26202:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1063.011763] Lustre: 26202:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1063.026474] Lustre: 26202:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1063.038868] Lustre: 26202:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1063.047914] Lustre: 26202:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1063.062720] Lustre: 26202:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1063.076705] Lustre: 26202:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1070.012201] Lustre: *** cfs_fail_loc=161d, val=0*** [ 1078.954662] Lustre: Failing over lustre-MDT0000 [ 1081.285959] Lustre: server umount lustre-MDT0000 complete [ 1091.934485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1092.053059] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1092.311476] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1096.285935] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1097.703722] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1097.709356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1097.728235] Lustre: Skipped 3 previous similar messages [ 1097.752490] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1097.782892] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 1097.796634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:489 to 0x280000401:513) [ 1104.143810] Lustre: DEBUG MARKER: == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 18:29:32 (1789684172) [ 1106.301526] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1115.071146] Lustre: DEBUG MARKER: == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 18:29:44 (1789684184) [ 1120.308470] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1121.033116] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1131.909398] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 18:30:00 (1789684200) [ 1166.464971] Lustre: Failing over lustre-MDT0000 [ 1166.877661] Lustre: server umount lustre-MDT0000 complete [ 1169.375873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1169.384147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.387589] LustreError: 26203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1169.396192] Lustre: Skipped 8 previous similar messages [ 1169.444700] LustreError: 26203:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 30 previous similar messages [ 1172.345482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1181.646914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1186.272078] Lustre: 16395:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789684240/real 1789684240] req@ffff8d08f4738a80 x1876619168558336/t0(0) o400->MGC192.168.203.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789684256 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1186.299501] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1194.028846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1194.051845] Lustre: lustre-MDT0000: reset Object Index mappings [ 1195.512276] LustreError: 16394:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8d08c56aad80 x1876619168566912/t0(0) o250->MGC192.168.203.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1195.854951] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1201.011165] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1201.135409] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1201.142252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1201.163660] Lustre: Skipped 3 previous similar messages [ 1201.220924] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1201.277469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:609) [ 1201.281300] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:609) [ 1204.344641] LustreError: 42866:0:(lfsck_engine.c:1018:lfsck_master_engine()) lustre-MDT0000-osd: master engine fail to verify the .lustre/lost+found/, go ahead: rc = -115 [ 1204.358812] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1208.484543] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1208.489373] Lustre: Skipped 3 previous similar messages [ 1213.032362] Lustre: Failing over lustre-MDT0000 [ 1213.328927] Lustre: server umount lustre-MDT0000 complete [ 1216.483105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1223.614453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1228.554988] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1229.348786] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:641) [ 1229.350157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:641) [ 1232.555345] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1240.213761] Lustre: DEBUG MARKER: == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 18:31:49 (1789684309) [ 1242.507857] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1251.390166] Lustre: Failing over lustre-MDT0000 [ 1251.676171] Lustre: server umount lustre-MDT0000 complete [ 1254.883081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1256.877855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1268.478608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1271.263116] Lustre: 16397:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789684325/real 1789684325] req@ffff8d08f76f9f80 x1876619168654976/t0(0) o400->MGC192.168.203.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789684341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1280.717563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1280.754249] Lustre: lustre-MDT0000: reset Object Index mappings [ 1295.841545] LustreError: 16394:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8d08f6f44380 x1876619168667392/t0(0) o250->MGC192.168.203.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1296.386962] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1296.404325] Lustre: Skipped 1 previous similar message [ 1301.473494] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1301.477030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1301.485662] Lustre: Skipped 1 previous similar message [ 1301.506720] Lustre: Skipped 7 previous similar messages [ 1301.539592] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1301.549050] Lustre: Skipped 1 previous similar message [ 1301.587911] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:705) [ 1301.588448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:705) [ 1301.618435] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1305.404708] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1320.620500] Lustre: Failing over lustre-MDT0000 [ 1321.098825] Lustre: server umount lustre-MDT0000 complete [ 1321.952299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1321.952309] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.985335] Lustre: Skipped 10 previous similar messages [ 1331.546063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1331.626607] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1331.635429] LustreError: Skipped 2 previous similar messages [ 1331.858743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1331.864712] Lustre: Skipped 4 previous similar messages [ 1337.000932] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1337.385152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:737) [ 1337.385155] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:737) [ 1340.321440] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1340.327913] Lustre: Skipped 84 previous similar messages [ 1348.079191] Lustre: DEBUG MARKER: == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 18:33:36 (1789684416) [ 1349.257255] Lustre: 27725:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 278, rollback = 2 [ 1349.268449] Lustre: 27725:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1306 previous similar messages [ 1349.279356] Lustre: 27725:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 1349.293697] Lustre: 27725:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1306 previous similar messages [ 1349.308936] Lustre: 27725:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1349.322628] Lustre: 27725:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1306 previous similar messages [ 1349.334418] Lustre: 27725:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1349.349067] Lustre: 27725:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1306 previous similar messages [ 1349.356708] Lustre: 27725:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 1349.361295] Lustre: 27725:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1306 previous similar messages [ 1349.367124] Lustre: 27725:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1349.372420] Lustre: 27725:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1306 previous similar messages [ 1356.370167] Lustre: *** cfs_fail_loc=1600, val=1*** [ 1356.377645] Lustre: Skipped 8 previous similar messages [ 1378.754118] Lustre: DEBUG MARKER: == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 18:34:07 (1789684447) [ 1389.343752] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1389.348017] Lustre: Skipped 12 previous similar messages [ 1408.923115] Lustre: DEBUG MARKER: == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 18:34:37 (1789684477) [ 1423.335558] Lustre: Failing over lustre-MDT0000 [ 1423.642329] Lustre: server umount lustre-MDT0000 complete [ 1424.353432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1429.471725] LustreError: 27725:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1429.484544] LustreError: 27725:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 87 previous similar messages [ 1432.269433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1432.662585] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1432.668225] Lustre: Skipped 1 previous similar message [ 1436.524759] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1437.676405] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1437.684697] Lustre: Skipped 1 previous similar message [ 1437.687732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1437.696634] Lustre: Skipped 7 previous similar messages [ 1437.716775] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1437.727048] Lustre: Skipped 1 previous similar message [ 1437.777112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:854 to 0x280000401:897) [ 1437.777822] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:853 to 0x2c0000401:897) [ 1445.432200] Lustre: DEBUG MARKER: == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 18:35:13 (1789684513) [ 1458.298709] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 1472.862204] Lustre: 52933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1492.221762] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1494.999599] Lustre: 54076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1501.175973] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1501.178021] Lustre: Skipped 81 previous similar messages [ 1503.367417] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1504.415168] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1505.441857] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1507.487547] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1507.489653] Lustre: Skipped 1 previous similar message [ 1507.501213] Lustre: Failing over lustre-MDT0000 [ 1507.695562] Lustre: server umount lustre-MDT0000 complete [ 1515.888736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1519.709524] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1521.723133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:946 to 0x280000401:961) [ 1521.731088] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:947 to 0x2c0000401:993) [ 1527.322830] Lustre: DEBUG MARKER: == sanity-lfsck test 8: LFSCK state machine ============== 18:36:36 (1789684596) [ 1531.874776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1535.150721] Lustre: server umount lustre-MDT0000 complete [ 1538.151433] LustreError: 26189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684608 with bad export cookie 732587196721272224 [ 1538.163894] LustreError: 26189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1538.359310] Lustre: server umount lustre-MDT0001 complete [ 1551.391817] Lustre: server umount lustre-OST0000 complete [ 1563.702351] Lustre: server umount lustre-OST0001 complete [ 1569.502210] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 1576.119689] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 1609.104414] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 1617.715170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.865302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1617.889550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1617.960948] Lustre: lustre-MDT0000: new disk, initializing [ 1618.033741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1621.465766] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1629.796909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.896922] Lustre: 59130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1629.929496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1629.933717] Lustre: Skipped 1 previous similar message [ 1630.002414] Lustre: lustre-MDT0001: new disk, initializing [ 1630.075886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1630.084532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1633.611797] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1637.586360] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1643.109992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1643.319828] Lustre: lustre-OST0000: new disk, initializing [ 1643.326674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1643.337151] Lustre: 60759:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1644.622476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1644.635670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1644.674572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1648.410829] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1656.665284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1656.764378] Lustre: lustre-OST0001: new disk, initializing [ 1656.769656] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1656.776814] Lustre: 61630:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1658.000084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1658.007233] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1658.067062] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1662.099237] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1669.165164] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1671.167981] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1680.064820] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1680.762963] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1680.767082] Lustre: Skipped 19 previous similar messages [ 1683.944455] Lustre: *** cfs_fail_loc=1601, val=2*** [ 1683.949979] Lustre: Skipped 15 previous similar messages [ 1698.295450] Lustre: Failing over lustre-MDT0000 [ 1698.472181] Lustre: server umount lustre-MDT0000 complete [ 1699.300199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.311182] Lustre: Skipped 14 previous similar messages [ 1705.477092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1705.599332] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1705.612213] LustreError: Skipped 3 previous similar messages [ 1705.866822] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1705.879727] Lustre: Skipped 1 previous similar message [ 1709.089985] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1711.076417] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1711.079077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1711.081995] Lustre: Skipped 1 previous similar message [ 1711.089590] Lustre: Skipped 7 previous similar messages [ 1711.107639] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1711.114019] Lustre: Skipped 1 previous similar message [ 1711.136278] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1711.143629] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:65) [ 1711.152391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:65) [ 1715.272206] Lustre: Failing over lustre-MDT0000 [ 1715.504386] Lustre: server umount lustre-MDT0000 complete [ 1723.019718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1726.543612] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1728.518393] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1728.523225] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:97) [ 1728.523302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:97) [ 1730.456526] Lustre: Failing over lustre-MDT0000 [ 1730.649671] Lustre: server umount lustre-MDT0000 complete [ 1733.604140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1733.610531] LustreError: Skipped 1 previous similar message [ 1736.651693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1740.137669] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1742.365672] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:129) [ 1742.367111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:129) [ 1744.279673] Lustre: *** cfs_fail_loc=1602, val=2*** [ 1751.854473] Lustre: DEBUG MARKER: == sanity-lfsck test 9a: LFSCK speed control (1) ========= 18:40:21 (1789684821) [ 1761.440758] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 1773.297388] Lustre: 68533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1789.450546] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1791.885676] Lustre: 69668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1882.190942] Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) ========= 18:42:31 (1789684951) [ 1912.279660] Lustre: 59135:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 1912.284891] Lustre: 59135:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 17765 previous similar messages [ 1912.288910] Lustre: 59135:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1912.293472] Lustre: 59135:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 17765 previous similar messages [ 1912.302083] Lustre: 59135:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1912.307236] Lustre: 59135:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 17765 previous similar messages [ 1912.313402] Lustre: 59135:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 1912.318160] Lustre: 59135:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 17765 previous similar messages [ 1912.323743] Lustre: 59135:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 1912.328336] Lustre: 59135:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 17765 previous similar messages [ 1912.333351] Lustre: 59135:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1912.337847] Lustre: 59135:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 17765 previous similar messages [ 1916.564508] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1916.569605] Lustre: Skipped 4 previous similar messages [ 1933.438887] Lustre: *** cfs_fail_loc=160c, val=0*** [ 1933.441424] Lustre: Skipped 6 previous similar messages [ 1962.782329] Lustre: DEBUG MARKER: == sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 18:43:52 (1789685032) [ 1995.044389] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2003.050322] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2003.054789] Lustre: Skipped 643 previous similar messages [ 2019.055996] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2019.059990] Lustre: Skipped 1206 previous similar messages [ 2044.626930] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2044.629872] Lustre: Skipped 3105 previous similar messages [ 2219.200807] Lustre: DEBUG MARKER: == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 18:48:08 (1789685288) [ 2361.823780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2361.824486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.825397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.825403] Lustre: Skipped 3 previous similar messages [ 2361.865158] Lustre: Skipped 12 previous similar messages [ 2363.386951] Lustre: server umount lustre-MDT0000 complete [ 2366.712869] LustreError: 59122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685437 with bad export cookie 732587196721291327 [ 2366.722964] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.734064] LustreError: Skipped 2 previous similar messages [ 2366.943868] LustreError: 59135:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.953232] LustreError: 59135:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 32 previous similar messages [ 2367.080498] Lustre: server umount lustre-MDT0001 complete [ 2376.636232] Lustre: server umount lustre-OST0000 complete [ 2380.016939] Lustre: server umount lustre-OST0001 complete [ 2385.664741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 2392.455282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2407.967469] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2413.087939] LustreError: 75379:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.146@tcp: failed processing log, type 4: rc = -110 [ 2438.757083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2438.762730] Lustre: Skipped 9 previous similar messages [ 2443.485694] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2446.339204] Lustre: 75963:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2446.363305] Lustre: *** cfs_fail_loc=160e, val=3*** [ 2449.380243] Lustre: 75963:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2455.941530] Lustre: DEBUG MARKER: == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 18:52:04 (1789685524) [ 2468.109871] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 2477.994124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.426775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3272 to 0x280000401:3297) [ 2482.341344] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2489.003917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.279526] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2495.881909] Lustre: 78627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2507.425609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2512.030272] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2512.912308] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3207 to 0x2c0000401:3233) [ 2518.094719] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2520.945912] Lustre: 80125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2521.971483] Lustre: 77478:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 2521.981747] Lustre: 77478:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 22021 previous similar messages [ 2521.986051] Lustre: 77478:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 2521.991145] Lustre: 77478:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 22021 previous similar messages [ 2521.999171] Lustre: 77478:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 2522.005138] Lustre: 77478:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 22020 previous similar messages [ 2522.014587] Lustre: 77478:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 2522.019562] Lustre: 77478:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 22021 previous similar messages [ 2522.024923] Lustre: 77478:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 2522.030728] Lustre: 77478:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 22021 previous similar messages [ 2522.036892] Lustre: 77478:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 2522.044455] Lustre: 77478:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 22021 previous similar messages [ 2524.659775] Lustre: *** cfs_fail_loc=160d, val=0*** [ 2530.117534] Lustre: Failing over lustre-OST0000 [ 2530.206411] Lustre: server umount lustre-OST0000 complete [ 2539.817553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2540.061677] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2540.075455] Lustre: Skipped 2 previous similar messages [ 2541.538934] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2541.550861] Lustre: Skipped 2 previous similar messages [ 2541.587565] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2541.589102] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2541.589739] Lustre: *** cfs_fail_loc=215, val=0*** [ 2541.615035] Lustre: Skipped 2 previous similar messages [ 2541.644556] Lustre: Skipped 11 previous similar messages [ 2546.656345] Lustre: *** cfs_fail_loc=215, val=0*** [ 2546.672562] Lustre: Skipped 3 previous similar messages [ 2548.006982] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2551.779564] Lustre: *** cfs_fail_loc=215, val=0*** [ 2553.353695] Lustre: 81526:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2553.363713] Lustre: 81526:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2556.239165] Lustre: Failing over lustre-OST0000 [ 2556.599588] Lustre: server umount lustre-OST0000 complete [ 2567.104600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2569.284741] Lustre: *** cfs_fail_loc=215, val=0*** [ 2569.293112] Lustre: Skipped 1 previous similar message [ 2574.304406] Lustre: *** cfs_fail_loc=215, val=0*** [ 2574.310958] Lustre: Skipped 4 previous similar messages [ 2574.966796] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2583.028257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2583.043985] Lustre: Skipped 6 previous similar messages [ 2587.997215] Lustre: server umount lustre-MDT0000 complete [ 2593.170548] LustreError: 75386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685663 with bad export cookie 732587196722850969 [ 2593.193872] LustreError: 75386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2593.261826] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2593.267107] Lustre: Skipped 1 previous similar message [ 2593.502928] Lustre: server umount lustre-MDT0001 complete [ 2601.234861] Lustre: server umount lustre-OST0000 complete [ 2606.348208] Lustre: server umount lustre-OST0001 complete [ 2618.638278] Lustre: DEBUG MARKER: == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 18:54:47 (1789685687) [ 2640.639313] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 2653.510281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.333792] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2668.407232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2673.772222] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2677.058877] Lustre: 85912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2686.228489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.752267] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2699.772442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3362 to 0x280000401:3393) [ 2707.071666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2712.576169] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3207 to 0x2c0000401:3265) [ 2713.340769] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2721.943655] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2725.909512] Lustre: 87784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2768.340455] Lustre: DEBUG MARKER: == sanity-lfsck test 12b: auto detect Lustre device ====== 18:57:16 (1789685836) [ 2786.595491] Lustre: DEBUG MARKER: == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 18:57:35 (1789685855) [ 2788.146890] Lustre: *** cfs_fail_loc=160f, val=0*** [ 2800.203425] Lustre: DEBUG MARKER: == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 18:57:49 (1789685869) [ 2803.935682] Lustre: *** cfs_fail_loc=1610, val=0*** [ 2803.937744] Lustre: Skipped 7 previous similar messages [ 2852.832581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2858.579276] Lustre: server umount lustre-MDT0000 complete [ 2862.132974] LustreError: 84752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685932 with bad export cookie 732587196722859474 [ 2862.392130] Lustre: server umount lustre-MDT0001 complete [ 2876.634424] Lustre: server umount lustre-OST0000 complete [ 2890.831433] Lustre: server umount lustre-OST0001 complete [ 2906.879619] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 2917.129296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.613190] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2931.049828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.118748] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2939.204630] Lustre: 93651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2945.650242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2952.725746] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2956.337509] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:161) [ 2962.418714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3554 to 0x280000401:3585) [ 2962.594368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2968.060148] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:161) [ 2968.060149] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3361) [ 2968.963459] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2976.062678] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2979.232573] Lustre: 95519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2986.023815] Lustre: DEBUG MARKER: == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 19:00:54 (1789686054) [ 2990.616096] Lustre: *** cfs_fail_loc=1610, val=0*** [ 2990.618068] Lustre: Skipped 63 previous similar messages [ 3018.721036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3018.736579] LustreError: Skipped 4 previous similar messages [ 3018.742983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.767495] Lustre: Skipped 17 previous similar messages [ 3018.777038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3018.794246] Lustre: Skipped 4 previous similar messages [ 3021.788397] Lustre: server umount lustre-MDT0000 complete [ 3024.354940] LustreError: 92512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.379216] LustreError: 92512:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 58 previous similar messages [ 3025.607368] LustreError: 92492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686096 with bad export cookie 732587196722887894 [ 3025.620937] LustreError: 92492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3025.640856] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3025.658760] LustreError: Skipped 2 previous similar messages [ 3025.914864] Lustre: server umount lustre-MDT0001 complete [ 3040.177827] Lustre: server umount lustre-OST0000 complete [ 3054.595215] Lustre: server umount lustre-OST0001 complete [ 3078.455475] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 3091.021866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3091.697180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3091.712491] Lustre: Skipped 13 previous similar messages [ 3096.584974] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3109.142470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.687734] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3121.820347] Lustre: 99569:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3132.286835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3134.635700] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:193) [ 3136.698523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3682 to 0x280000401:3713) [ 3140.474476] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3149.465669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3154.920300] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3393) [ 3154.921656] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:193) [ 3155.823895] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3162.811807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3171.727944] Lustre: DEBUG MARKER: == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 19:04:00 (1789686240) [ 3173.712863] Lustre: 100746:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 3173.717844] Lustre: 100746:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1496 previous similar messages [ 3173.722809] Lustre: 100746:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 3173.728643] Lustre: 100746:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1496 previous similar messages [ 3173.735330] Lustre: 100746:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3173.740748] Lustre: 100746:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1496 previous similar messages [ 3173.743072] Lustre: 100746:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3173.750795] Lustre: 100746:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1496 previous similar messages [ 3173.757327] Lustre: 100746:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 3173.762709] Lustre: 100746:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1496 previous similar messages [ 3173.769519] Lustre: 100746:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3173.775447] Lustre: 100746:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1496 previous similar messages [ 3175.143584] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3175.146819] Lustre: Skipped 63 previous similar messages [ 3175.332895] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3183.820954] Lustre: DEBUG MARKER: == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 19:04:12 (1789686252) [ 3185.579668] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3185.618432] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3185.623962] Lustre: Skipped 3 previous similar messages [ 3194.769505] Lustre: DEBUG MARKER: == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 19:04:23 (1789686263) [ 3196.267932] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_15c MDS newer than 2.7.55, LU-6475 [ 3197.653292] Lustre: DEBUG MARKER: == sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 19:04:26 (1789686266) [ 3203.166936] Lustre: *** cfs_fail_loc=1709, val=0*** [ 3203.174406] LustreError: 98436:0:(mdt_reint.c:2644:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240002340:0x6c:0x0]/f20 failed: rc = -5 [ 3267.553777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3267.567851] Lustre: Skipped 7 previous similar messages [ 3271.779967] Lustre: server umount lustre-MDT0000 complete [ 3284.299376] LustreError: 102942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686354 with bad export cookie 732587196722902629 [ 3284.318231] LustreError: 102942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3284.966133] Lustre: server umount lustre-MDT0001 complete [ 3304.223947] Lustre: 16397:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789686358/real 1789686358] req@ffff8d07cef47480 x1876619173801216/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789686374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3305.247139] Lustre: 16396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789686359/real 1789686359] req@ffff8d07c9f89f80 x1876619173801472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789686375 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3307.523490] Lustre: server umount lustre-OST0000 complete [ 3308.450192] Lustre: 16398:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789686363/real 1789686363] req@ffff8d07cef44380 x1876619173801728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789686379 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3314.655098] Lustre: 16398:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789686369/real 1789686369] req@ffff8d07cc84ad80 x1876619173802368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789686385 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3314.694167] Lustre: 16398:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3316.609575] Lustre: server umount lustre-OST0001 complete [ 3334.166642] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 3336.897798] Key type lgssc unregistered [ 3337.211524] LNet: 105266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3337.219432] LNetError: 105266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3337.240446] LNet: Removed LNI 192.168.203.146@tcp [ 3338.274257] Key type .llcrypt unregistered [ 3338.280903] Key type ._llcrypt unregistered [ 3363.408965] Key type ._llcrypt registered [ 3363.417376] Key type .llcrypt registered [ 3363.603725] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 3376.013512] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 3376.744817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3376.774350] alg: No test for adler32 (adler32-zlib) [ 3377.873551] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 3378.194678] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 3379.841325] Key type lgssc registered [ 3381.071507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3428.389990] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 3442.893393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3442.924027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3444.189843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3444.216822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3444.303533] Lustre: lustre-MDT0000: new disk, initializing [ 3444.426883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3444.464324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3450.093912] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3469.453586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.736250] Lustre: 109695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3469.887860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3469.904958] Lustre: Skipped 1 previous similar message [ 3470.116158] Lustre: lustre-MDT0001: new disk, initializing [ 3470.291878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3470.379226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3470.418223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3478.299529] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3485.274874] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3495.260667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3495.402450] Lustre: lustre-OST0000: new disk, initializing [ 3495.409778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3495.415730] Lustre: 111638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3495.468713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3502.287576] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3505.201646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3505.218490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3505.341420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3516.661442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3516.785528] Lustre: lustre-OST0001: new disk, initializing [ 3516.789344] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3516.796474] Lustre: 112664:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3516.863365] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3522.518216] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3526.673204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3526.682947] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3526.732399] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3533.492977] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3540.253499] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3545.631859] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 19:10:14 (1789686614) === [ 3552.668842] Lustre: DEBUG MARKER: == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 19:10:21 (1789686621) [ 3552.880064] Lustre: 109702:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 3552.886635] Lustre: 109702:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3552.906792] Lustre: 109702:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3552.917357] Lustre: 109702:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 3552.921952] Lustre: 109702:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 3552.929499] Lustre: 109702:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3553.448613] Lustre: 109702:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 3553.460776] Lustre: 109702:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 9 previous similar messages [ 3553.466765] Lustre: 109702:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3553.480056] Lustre: 109702:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3553.489250] Lustre: 109702:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3553.498792] Lustre: 109702:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3553.503726] Lustre: 109702:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 7/369/2 [ 3553.508224] Lustre: 109702:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3553.514381] Lustre: 109702:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 3553.519142] Lustre: 109702:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3553.525193] Lustre: 109702:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3553.530844] Lustre: 109702:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3554.451963] Lustre: 109702:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 3554.459066] Lustre: 109702:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 185 previous similar messages [ 3554.469931] Lustre: 109702:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 3554.475706] Lustre: 109702:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 185 previous similar messages [ 3554.495583] Lustre: 109703:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3554.509250] Lustre: 109703:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 188 previous similar messages [ 3554.516750] Lustre: 109703:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 7/369/0 [ 3554.533930] Lustre: 109703:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 188 previous similar messages [ 3554.545320] Lustre: 109703:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 3554.552025] Lustre: 109703:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 188 previous similar messages [ 3554.559125] Lustre: 109703:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3554.567269] Lustre: 109703:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 188 previous similar messages [ 3556.087911] Lustre: *** cfs_fail_loc=1613, val=0*** [ 3565.410780] Lustre: DEBUG MARKER: == sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 19:10:34 (1789686634) [ 3566.517342] Lustre: 109703:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 3566.526663] Lustre: 109703:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 113 previous similar messages [ 3566.532583] Lustre: 109703:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3566.539526] Lustre: 109703:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 113 previous similar messages [ 3566.546938] Lustre: 109703:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3566.553114] Lustre: 109703:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 110 previous similar messages [ 3566.568056] Lustre: 109703:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 3566.586041] Lustre: 109703:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 110 previous similar messages [ 3566.601581] Lustre: 109703:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 3566.612218] Lustre: 109703:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 110 previous similar messages [ 3566.618963] Lustre: 109703:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3566.625780] Lustre: 109703:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 110 previous similar messages [ 3567.902919] Lustre: *** cfs_fail_loc=1614, val=0*** [ 3571.907972] Lustre: 111629:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 275, rollback = 2 [ 3571.922726] Lustre: 111629:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 11 previous similar messages [ 3571.927131] Lustre: 111629:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3571.932137] Lustre: 111629:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3571.937711] Lustre: 111629:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 3/275/0 [ 3571.942032] Lustre: 111629:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3571.946971] Lustre: 111629:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 1/4/0, quota 7/225/0 [ 3571.951736] Lustre: 111629:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3571.956535] Lustre: 111629:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 3571.960572] Lustre: 111629:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3571.965548] Lustre: 111629:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3571.969328] Lustre: 111629:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3578.545962] Lustre: DEBUG MARKER: == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 19:10:47 (1789686647) [ 3580.719827] Lustre: *** cfs_fail_loc=1615, val=0*** [ 3580.721781] Lustre: Skipped 1 previous similar message [ 3581.725293] Lustre: *** cfs_fail_loc=1615, val=0*** [ 3581.729618] Lustre: Skipped 3 previous similar messages [ 3598.122757] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_18b skipping excluded test 18b [ 3599.823743] Lustre: DEBUG MARKER: == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 19:11:08 (1789686668) [ 3600.209137] Lustre: 109702:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 3600.214618] Lustre: 109702:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 23 previous similar messages [ 3600.220742] Lustre: 109702:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 3600.228888] Lustre: 109702:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 3600.234519] Lustre: 109702:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3600.240494] Lustre: 109702:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 3600.245511] Lustre: 109702:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3600.252326] Lustre: 109702:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 3600.258139] Lustre: 109702:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 3600.265284] Lustre: 109702:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 3600.273832] Lustre: 109702:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3600.279740] Lustre: 109702:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 3602.199266] Lustre: *** cfs_fail_loc=1617, val=0*** [ 3602.271189] Lustre: *** cfs_fail_loc=1617, val=0*** [ 3603.999370] Lustre: *** cfs_fail_loc=1616, val=0*** [ 3604.003464] Lustre: Skipped 1 previous similar message [ 3619.715893] Lustre: DEBUG MARKER: == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 19:11:28 (1789686688) [ 3620.092878] Lustre: 109704:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 3620.103065] Lustre: 109704:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 24 previous similar messages [ 3620.108702] Lustre: 109704:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 3620.117510] Lustre: 109704:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 3620.125185] Lustre: 109704:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3620.135398] Lustre: 109704:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 3620.143361] Lustre: 109704:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3620.149124] Lustre: 109704:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 3620.155593] Lustre: 109704:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 3620.159349] Lustre: 109704:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 3620.164374] Lustre: 109704:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3620.169825] Lustre: 109704:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 3621.467722] Lustre: *** cfs_fail_loc=1618, val=0*** [ 3621.472665] Lustre: Skipped 5 previous similar messages [ 3654.623952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.624964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3654.639523] Lustre: Skipped 3 previous similar messages [ 3654.645086] Lustre: Skipped 2 previous similar messages [ 3659.745317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3659.748019] Lustre: Skipped 4 previous similar messages [ 3662.611087] Lustre: server umount lustre-MDT0000 complete [ 3665.904170] LustreError: 109689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686736 with bad export cookie 1029414106803757339 [ 3665.905601] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3665.913017] LustreError: 109689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3666.173290] Lustre: server umount lustre-MDT0001 complete [ 3680.307289] Lustre: server umount lustre-OST0000 complete [ 3694.224832] Lustre: server umount lustre-OST0001 complete [ 3708.763184] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 3716.235891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.575965] LustreError: 118367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3716.589103] LustreError: 118367:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3716.620485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3719.845853] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3721.697943] LustreError: 118368:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3725.664429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3725.886652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3728.729354] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3730.601518] Lustre: 119505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3735.300540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3736.482057] LustreError: 119859:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3736.498667] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:33) [ 3739.672719] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3741.603884] LustreError: 119860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.617140] LustreError: 119860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3741.617364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:116 to 0x280000401:161) [ 3745.213603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3745.339154] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3745.345483] Lustre: Skipped 1 previous similar message [ 3750.228554] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3750.374966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:33) [ 3750.381892] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 3755.718956] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3757.914834] Lustre: 121375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3766.710226] Lustre: DEBUG MARKER: == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 19:13:55 (1789686835) [ 3766.985020] Lustre: 118362:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 3767.000992] Lustre: 118362:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 21 previous similar messages [ 3767.013924] Lustre: 118362:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3767.023465] Lustre: 118362:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3767.030641] Lustre: 118362:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3767.037213] Lustre: 118362:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3767.044502] Lustre: 118362:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 3767.048565] Lustre: 118362:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3767.052759] Lustre: 118362:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 3767.057414] Lustre: 118362:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3767.061633] Lustre: 118362:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3767.066118] Lustre: 118362:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3768.392753] Lustre: *** cfs_fail_loc=1618, val=0*** [ 3768.395309] Lustre: Skipped 3 previous similar messages [ 3801.570713] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.571455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3801.583855] Lustre: Skipped 2 previous similar messages [ 3801.594198] Lustre: Skipped 3 previous similar messages [ 3806.690732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3806.700316] Lustre: Skipped 3 previous similar messages [ 3806.916914] Lustre: server umount lustre-MDT0000 complete [ 3809.718379] LustreError: 118347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686880 with bad export cookie 1029414106803772585 [ 3809.723501] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3809.731436] LustreError: 118347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3809.953397] Lustre: server umount lustre-MDT0001 complete [ 3823.361625] Lustre: server umount lustre-OST0000 complete [ 3835.619056] Lustre: server umount lustre-OST0001 complete [ 3847.517263] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 3855.310322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.707351] LustreError: 123945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3855.775273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3859.350538] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3866.001131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.080092] LustreError: 123946:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3866.094572] LustreError: 123946:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3869.809650] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3872.035896] Lustre: 125083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3876.965366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3881.252138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 3881.764492] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3886.560285] LustreError: 125437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3886.573088] LustreError: 125437:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3887.914356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3891.128684] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:65) [ 3891.129324] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:65) [ 3891.147469] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 3892.611326] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3898.164189] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3900.699602] Lustre: 126952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3904.510906] Lustre: 123946:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 252 < left 260, rollback = 2 [ 3904.519083] Lustre: 123946:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 21 previous similar messages [ 3904.527831] Lustre: 123946:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3904.533641] Lustre: 123946:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3904.537565] Lustre: 123946:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/260/0 [ 3904.541343] Lustre: 123946:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3904.545188] Lustre: 123946:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 0/0/0, quota 4/150/2 [ 3904.549284] Lustre: 123946:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3904.553308] Lustre: 123946:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 1/17/2, delete: 0/0/0 [ 3904.556963] Lustre: 123946:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3904.563202] Lustre: 123946:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3904.567036] Lustre: 123946:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3904.604229] Lustre: *** cfs_fail_loc=1602, val=10*** [ 3904.610151] Lustre: Skipped 1 previous similar message [ 3925.951656] Lustre: DEBUG MARKER: == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 19:16:35 (1789686995) [ 3928.110386] Lustre: *** cfs_fail_loc=1616, val=0*** [ 3928.112278] Lustre: Skipped 3 previous similar messages [ 3933.531708] Lustre: *** cfs_fail_loc=161c, val=0*** [ 3947.859206] Lustre: DEBUG MARKER: == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) ========================================================== 19:16:57 (1789687017) [ 3949.724895] Lustre: *** cfs_fail_loc=162e, val=0*** [ 3958.990393] Lustre: DEBUG MARKER: == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 19:17:08 (1789687028) [ 3962.173254] Lustre: *** cfs_fail_loc=162f, val=0*** [ 3962.176945] Lustre: Skipped 9 previous similar messages [ 3972.472457] Lustre: DEBUG MARKER: == sanity-lfsck test 19a: OST-object inconsistency self detect ========================================================== 19:17:21 (1789687041) [ 3980.704147] Lustre: DEBUG MARKER: == sanity-lfsck test 19b: OST-object inconsistency self repair ========================================================== 19:17:29 (1789687049) [ 3982.315270] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3982.334048] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3982.337106] Lustre: Skipped 3 previous similar messages [ 3985.595432] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.46@tcp inode [0x2000013a1:0x1f:0x0] object 0x280000401:203 extent [0-4095], client returned csum 0 (type 4), server csum 703fb3ad (type 4) [ 3986.622590] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.46@tcp inode [0x2000013a1:0x20:0x0] object 0x280000401:204 extent [0-4095], client returned csum 0 (type 4), server csum 7cbb935c (type 4) [ 3991.012317] Lustre: DEBUG MARKER: == sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================================================== 19:17:40 (1789687060) [ 4008.234405] Lustre: DEBUG MARKER: == sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case ========================================================== 19:17:57 (1789687077) [ 4012.950757] Lustre: DEBUG MARKER: == sanity-lfsck test 21: run all LFSCK components by default ========================================================== 19:18:02 (1789687082) [ 4021.130962] Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) ========================================================== 19:18:10 (1789687090) [ 4022.864170] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4022.875785] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4022.880384] Lustre: Skipped 1 previous similar message [ 4030.604237] Lustre: DEBUG MARKER: == sanity-lfsck test 22b: LFSCK can repair unmatched pairs (2) ========================================================== 19:18:19 (1789687099) [ 4031.767175] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4031.770158] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4039.167403] Lustre: DEBUG MARKER: == sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) ========================================================== 19:18:28 (1789687108) [ 4039.508861] Lustre: 126717:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 4039.520773] Lustre: 126717:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 462 previous similar messages [ 4039.528802] Lustre: 126717:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 4039.535938] Lustre: 126717:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 462 previous similar messages [ 4039.545725] Lustre: 126717:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4039.552625] Lustre: 126717:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 462 previous similar messages [ 4039.558823] Lustre: 126717:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4039.563675] Lustre: 126717:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 462 previous similar messages [ 4039.567867] Lustre: 126717:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 4039.573781] Lustre: 126717:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 462 previous similar messages [ 4039.578278] Lustre: 126717:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4039.582548] Lustre: 126717:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 462 previous similar messages [ 4040.404415] Lustre: *** cfs_fail_loc=1620, val=0*** [ 4041.617765] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 [ 4041.624252] PGD 0 P4D 0 [ 4041.631138] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [ 4041.636032] CPU: 2 PID: 133571 Comm: lfsck Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 4041.646185] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4041.653097] RIP: 0010:osd_iit_iget+0x67c/0x720 [osd_ldiskfs] [ 4041.656238] Code: ee fe 48 c7 c7 60 dd 90 c1 48 83 05 85 6c 06 00 01 e8 98 4c ee fe 48 83 05 e8 77 06 00 01 e9 eb fb ff ff 4c 8b 83 80 4c 01 00 <49> 3b 38 74 4b 48 8b 05 30 78 06 00 48 8d 48 01 31 c0 eb 11 48 63 [ 4041.669302] RSP: 0018:ffff9cbc0f2c7be0 EFLAGS: 00010246 [ 4041.673919] RAX: 0000000000000000 RBX: ffff8d07cfb00000 RCX: 0000000000000000 [ 4041.683251] RDX: ffff8d07d250d680 RSI: 0000000000000003 RDI: 0000000200000003 [ 4041.688327] RBP: ffff8d07d250d738 R08: 0000000000000000 R09: 0000000000000000 [ 4041.695382] R10: ffffffffc176d3b0 R11: ffffffffc18e3549 R12: 0000000000000002 [ 4041.704539] R13: 0000000000000000 R14: ffff8d07c6dec260 R15: 0000000000004e65 [ 4041.715127] FS: 0000000000000000(0000) GS:ffff8d0902100000(0000) knlGS:0000000000000000 [ 4041.723323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4041.731282] CR2: 0000000000000000 CR3: 0000000101e16002 CR4: 0000000000170ee0 [ 4041.738241] Call Trace: [ 4041.739154] ? show_regs.cold.9+0x22/0x2f [ 4041.744268] ? __die_body+0x22/0x90 [ 4041.747054] ? __die+0x33/0x4a [ 4041.749254] ? no_context+0x30f/0x5a0 [ 4041.752811] ? __bad_area_nosemaphore+0x1c6/0x260 [ 4041.757989] ? bad_area_nosemaphore+0x1a/0x30 [ 4041.762068] ? do_user_addr_fault+0x540/0x8a0 [ 4041.765869] ? _raw_spin_unlock+0x12/0x30 [ 4041.772949] ? __do_page_fault+0x6b/0xa0 [ 4041.777946] ? do_page_fault+0x87/0x30f [ 4041.779966] ? page_fault+0x1e/0x30 [ 4041.782056] ? ldiskfs_xattr_trusted_set+0x40/0x40 [ldiskfs] [ 4041.787514] ? osd_iit_iget+0x67c/0x720 [osd_ldiskfs] [ 4041.789538] osd_preload_next+0xa5/0xe0 [osd_ldiskfs] [ 4041.795568] osd_inode_iteration+0x470/0x1150 [osd_ldiskfs] [ 4041.798729] ? osd_iit_iget+0x720/0x720 [osd_ldiskfs] [ 4041.800804] ? osd_obj_spec_lookup+0x410/0x410 [osd_ldiskfs] [ 4041.808618] ? do_raw_spin_unlock+0x75/0x190 [ 4041.812372] ? do_raw_spin_unlock+0x75/0x190 [ 4041.818031] osd_otable_it_next+0x223/0x8f0 [osd_ldiskfs] [ 4041.826269] ? init_wait_var_entry+0x50/0x50 [ 4041.834258] lfsck_master_oit_engine+0x3eb/0x1d00 [lfsck] [ 4041.838302] lfsck_master_engine+0x2f9/0x10d0 [lfsck] [ 4041.847729] ? __schedule+0x369/0xcb0 [ 4041.849216] ? do_raw_spin_unlock+0x75/0x190 [ 4041.851422] ? lfsck_prep+0xd70/0xd70 [lfsck] [ 4041.854185] kthread+0x1d1/0x200 [ 4041.857304] ? set_kthread_struct+0x70/0x70 [ 4041.859272] ret_from_fork+0x1f/0x30 [ 4041.860908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey zfs(O) spl(O) ec(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul ata_generic crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: libcfs] [ 4041.884808] CR2: 0000000000000000 [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-957.1.3.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Thu Nov 29 14:49:43 UTC 2018 [ 0.000000] Command line: rd.shell ro panic=1 nomodeset ipmtu=9000 rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr disable_cpu_apicid=0 acpi_rsdp=0xf52d0 elfcorehdr=3079540K [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ac000000-0x00000000bbf5cfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bbfffc00-0x00000000bbffffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] extended physical RAM map: [ 0.000000] reserve setup_data: [mem 0x0000000000000000-0x0000000000000fff] reserved [ 0.000000] reserve setup_data: [mem 0x0000000000001000-0x000000000009fbff] usable [ 0.000000] reserve setup_data: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000ac000000-0x00000000ac00002f] usable [ 0.000000] reserve setup_data: [mem 0x00000000ac000030-0x00000000bbf5cfff] usable [ 0.000000] reserve setup_data: [mem 0x00000000bbfffc00-0x00000000bbffffff] usable [ 0.000000] reserve setup_data: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0xbc000 max_arch_pfn = 0x400000000 [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] x2apic enabled by BIOS, switching to x2apic ops [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] mapped at [ffffffffff2004b0] [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0xb9421000-0xba7fffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f52d0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe247c 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe2318 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe238c 00090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe241c 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe2454 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] Setting APIC routing to cluster x2apic. [ 0.000000] NUMA turned off [ 0.000000] Faking a node at [mem 0x0000000000000000-0x00000000bbffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0xbbf36000-0xbbf5cfff] [ 0.000000] kvm-clock: cpu 0, msr 0:bbee5001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 4045006329067 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0xac000000-0xbbf5cfff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0xbbf5cfff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: Disabling requested cpu. Processor 0/0x0 ignored. [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: NR_CPUS/possible_cpus limit of 1 almost reached. Keeping one slot for boot cpu. Processor 1/0x1 ignored. [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: NR_CPUS/possible_cpus limit of 1 reached. Processor 3/0x3 ignored. [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: 4 Processors exceeds NR_CPUS limit of 1 [ 0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x00100000-0xabffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xac000000-0xac000fff] [ 0.000000] PM: Registered nosave memory: [mem 0xbbf5d000-0xbbffffff] [ 0.000000] e820: [mem 0x00100000-0xabffffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 38 pages/cpu @ffff8800bbc00000 s118784 r8192 d28672 u2097152 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr bbc1c040 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 64485 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: rd.shell ro panic=1 nomodeset ipmtu=9000 rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr disable_cpu_apicid=0 acpi_rsdp=0xf52d0 elfcorehdr=3079540K [ 0.000000] audit: disabled (until reboot) [ 0.000000] Misrouted IRQ fixup and polling support enabled [ 0.000000] This may significantly impact system performance [ 0.000000] Disabling memory control group subsystem [ 0.000000] PID hash table entries: 1024 (order: 1, 8192 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 213820k/3080192k available (7664k kernel code, 2818068k absent, 48304k reserved, 6055k data, 1876k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] x86/pti: Unmapping kernel while in userspace [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1. [ 0.000000] NR_IRQS:327936 nr_irqs:256 0 [ 0.000000] do_IRQ: 0.236 No irq handler for vector (irq -1) [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.374555] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.378756] pid_max: default: 32768 minimum: 301 [ 0.381915] Security Framework initialized [ 0.384098] SELinux: Initializing. [ 0.385669] Yama: becoming mindful. [ 0.387719] Dentry cache hash table entries: 32768 (order: 6, 262144 bytes) [ 0.392229] Inode-cache hash table entries: 16384 (order: 5, 131072 bytes) [ 0.395651] Mount-cache hash table entries: 512 (order: 0, 4096 bytes) [ 0.398476] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes) [ 0.403156] Initializing cgroup subsys memory [ 0.405157] Initializing cgroup subsys devices [ 0.406284] Initializing cgroup subsys freezer [ 0.407449] Initializing cgroup subsys net_cls [ 0.414156] Initializing cgroup subsys blkio [ 0.418672] Initializing cgroup subsys perf_event [ 0.421512] Initializing cgroup subsys hugetlb [ 0.423972] Initializing cgroup subsys pids [ 0.426439] Initializing cgroup subsys net_prio [ 0.428255] x86/cpu: Activated the Intel User Mode Instruction Prevention (UMIP) CPU feature [ 0.435294] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.439249] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.442681] tlb_flushall_shift: 6 [ 0.445231] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.452377] FEATURE SPEC_CTRL Present [ 0.454406] FEATURE IBPB_SUPPORT Present [ 0.457505] Spectre V2 : Mitigation: Full retpoline [ 0.503768] Freeing SMP alternatives: 28k freed [ 0.523018] ACPI: Core revision 20130517 [ 0.527680] ACPI: All ACPI Tables successfully acquired [ 0.535141] ftrace: allocating 29188 entries in 115 pages [ 0.632726] Switched APIC routing to physical x2apic. [ 0.641438] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.644893] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.649590] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.653122] ... version: 2 [ 0.654162] ... bit width: 48 [ 0.655241] ... generic registers: 4 [ 0.656612] ... value mask: 0000ffffffffffff [ 0.660480] ... max period: 00007fffffffffff [ 0.663580] ... fixed-purpose events: 3 [ 0.665744] ... event mask: 000000070000000f [ 0.668411] KVM setup paravirtual spinlock [ 0.673623] Brought up 1 CPUs [ 0.674669] smpboot: Max logical packages: 1 [ 0.678228] smpboot: Total of 1 processors activated (4799.99 BogoMIPS) [ 0.683198] devtmpfs: initialized [ 0.688543] EVM: security.selinux [ 0.690073] EVM: security.ima [ 0.691198] EVM: security.capability [ 0.695460] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.699390] pinctrl core: initialized pinctrl subsystem [ 0.704841] RTC time: 23:18:33, date: 09/17/26 [ 0.706485] NET: Registered protocol family 16 [ 0.710103] ACPI: bus type PCI registered [ 0.712261] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.716451] PCI: Using configuration type 1 for base access [ 0.718378] core: PMU erratum BJ122, BV98, HSD29 workaround disabled, HT off [ 0.724464] ACPI: Added _OSI(Module Device) [ 0.727042] ACPI: Added _OSI(Processor Device) [ 0.729317] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.731699] ACPI: Added _OSI(Processor Aggregator Device) [ 0.735312] ACPI: Added _OSI(Linux-Dell-Video) [ 0.740093] ACPI: Interpreter enabled [ 0.742328] ACPI: (supports S0 S3 S4 S5) [ 0.744275] ACPI: Using IOAPIC for interrupt routing [ 0.747477] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.753107] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.773226] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.780289] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.784433] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.790142] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.798403] acpiphp: Slot [2] registered [ 0.800107] acpiphp: Slot [5] registered [ 0.802970] acpiphp: Slot [6] registered [ 0.806041] acpiphp: Slot [7] registered [ 0.808907] acpiphp: Slot [8] registered [ 0.812037] acpiphp: Slot [9] registered [ 0.813610] acpiphp: Slot [10] registered [ 0.816902] acpiphp: Slot [3] registered [ 0.818467] acpiphp: Slot [4] registered [ 0.820719] acpiphp: Slot [11] registered [ 0.827127] acpiphp: Slot [12] registered [ 0.831133] acpiphp: Slot [13] registered [ 0.834169] acpiphp: Slot [14] registered [ 0.836064] acpiphp: Slot [15] registered [ 0.841458] acpiphp: Slot [16] registered [ 0.846761] acpiphp: Slot [17] registered [ 0.851456] acpiphp: Slot [18] registered [ 0.853695] acpiphp: Slot [19] registered [ 0.855394] acpiphp: Slot [20] registered [ 0.859689] acpiphp: Slot [21] registered [ 0.862863] acpiphp: Slot [22] registered [ 0.864952] acpiphp: Slot [23] registered [ 0.871616] acpiphp: Slot [24] registered [ 0.873647] acpiphp: Slot [25] registered [ 0.876932] acpiphp: Slot [26] registered [ 0.881044] acpiphp: Slot [27] registered [ 0.883053] acpiphp: Slot [28] registered [ 0.885615] acpiphp: Slot [29] registered [ 0.890079] acpiphp: Slot [30] registered [ 0.892185] acpiphp: Slot [31] registered [ 0.894036] PCI host bridge to bus 0000:00 [ 0.898662] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.902940] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.905598] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.910342] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.915989] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.921471] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.948619] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.954197] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.959397] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.963985] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.968978] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.972533] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 1.357970] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 10 *11) [ 1.360745] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.364392] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.367901] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 *10 11) [ 1.370288] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.372855] ACPI: NR_CPUS/possible_cpus limit of 1 reached. Processor 4/0x1 ignored. [ 1.376799] ACPI: Unable to map lapic to logical cpu number [ 1.382068] acpi LNXCPU:02: BIOS reported wrong ACPI id 0 for the processor [ 1.384888] ACPI: NR_CPUS/possible_cpus limit of 1 reached. Processor 5/0x3 ignored. [ 1.388330] ACPI: Unable to map lapic to logical cpu number [ 1.391002] vgaarb: loaded [ 1.392217] SCSI subsystem initialized [ 1.393997] ACPI: bus type USB registered [ 1.398718] usbcore: registered new interface driver usbfs [ 1.402095] usbcore: registered new interface driver hub [ 1.405496] usbcore: registered new device driver usb [ 1.407544] EDAC MC: Ver: 3.0.0 [ 1.409669] PCI: Using ACPI for IRQ routing [ 1.418393] NetLabel: Initializing [ 1.419813] NetLabel: domain hash size = 128 [ 1.421538] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.425624] NetLabel: unlabeled traffic allowed by default [ 1.431535] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.435450] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.441640] amd_nb: Cannot enumerate AMD northbridges [ 1.444769] Switched to clocksource kvm-clock [ 1.471005] pnp: PnP ACPI init [ 1.475553] ACPI: bus type PNP registered [ 1.481315] pnp: PnP ACPI: found 6 devices [ 1.482937] ACPI: bus type PNP unregistered [ 1.494173] NET: Registered protocol family 2 [ 1.497450] TCP established hash table entries: 2048 (order: 2, 16384 bytes) [ 1.503646] TCP bind hash table entries: 2048 (order: 3, 32768 bytes) [ 1.508023] TCP: Hash tables configured (established 2048 bind 2048) [ 1.515913] TCP: reno registered [ 1.517198] UDP hash table entries: 256 (order: 1, 8192 bytes) [ 1.519858] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes) [ 1.523948] NET: Registered protocol family 1 [ 1.527749] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.534745] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.537183] Unpacking initramfs... [ 5.585689] Freeing initrd memory: 20348k freed [ 5.596265] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 5.599830] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 5.602246] RAPL PMU: hw unit of domain package 2^-0 Joules [ 5.604384] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 5.607877] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 5.611665] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 5.617284] futex hash table entries: 256 (order: 2, 16384 bytes) [ 5.620446] Initialise system trusted keyring [ 5.668170] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 5.671361] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 5.675495] zpool: loaded [ 5.676796] zbud: loaded [ 5.678351] VFS: Disk quotas dquot_6.5.2 [ 5.680065] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 5.683042] msgmni has been set to 457 [ 5.684504] Key type big_key registered [ 5.687240] NET: Registered protocol family 38 [ 5.690204] Key type asymmetric registered [ 5.692254] Asymmetric key parser 'x509' registered [ 5.694610] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 5.697993] io scheduler noop registered [ 5.699390] io scheduler deadline registered (default) [ 5.701933] io scheduler cfq registered [ 5.704070] io scheduler mq-deadline registered [ 5.708453] io scheduler kyber registered [ 5.710339] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 5.712119] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 5.715094] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.718444] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.722002] ACPI: Power Button [PWRF] [ 5.723987] GHES: HEST is not enabled! [ 5.725815] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.778048] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 5.834241] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 5.840011] Non-volatile memory driver v1.3 [ 5.841821] Linux agpgart interface v0.103 [ 5.847339] crash memory driver: version 1.1 [ 5.848878] rdac: device handler registered [ 5.854539] hp_sw: device handler registered [ 5.859121] emc: device handler registered [ 5.860645] alua: device handler registered [ 5.862041] libphy: Fixed MDIO Bus: probed [ 5.870680] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 5.875728] ehci-pci: EHCI PCI platform driver [ 5.877306] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 5.879359] ohci-pci: OHCI PCI platform driver [ 5.881078] uhci_hcd: USB Universal Host Controller Interface driver [ 5.883592] usbcore: registered new interface driver usbserial_generic [ 5.886053] usbserial: USB Serial support registered for generic [ 5.888678] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.893178] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.896321] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.901688] mousedev: PS/2 mouse device common for all mice [ 5.905420] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.910423] rtc_cmos 00:05: RTC can wake from S4 [ 5.913588] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 5.917582] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 5.920456] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.924859] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.928838] cpuidle: using governor menu [ 5.931880] hidraw: raw HID events driver (C) Jiri Kosina [ 5.935116] usbcore: registered new interface driver usbhid [ 5.938612] usbhid: USB HID core driver [ 5.940506] drop_monitor: Initializing network drop monitor service [ 5.943142] TCP: cubic registered [ 5.944649] Initializing XFRM netlink socket [ 5.947002] NET: Registered protocol family 10 [ 5.949288] NET: Registered protocol family 17 [ 5.950984] mpls_gso: MPLS GSO support [ 5.953655] mce: Unable to init device /dev/mcelog (rc: -5) [ 5.957037] Loading compiled-in X.509 certificates [ 5.959662] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 5.965743] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 5.971861] Loaded X.509 cert 'CentOS Linux kernel signing key: e7cef3613a9b8bd012fae7498272159bb1879c65' [ 5.976094] registered taskstats version 1 [ 5.983447] Key type trusted registered [ 5.990284] Key type encrypted registered [ 5.991800] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 5.996667] Magic number: 10:10:353 [ 5.998268] rtc_cmos 00:05: setting system clock to 2026-09-17 23:18:39 UTC (1789687119) [ 6.003186] Freeing unused kernel memory: 1876k freed [ 6.006394] Write protecting the kernel read-only data: 12288k [ 6.011077] Freeing unused kernel memory: 516k freed [ 6.015924] Freeing unused kernel memory: 600k freed [ 6.034885] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.044237] systemd[1]: Detected virtualization kvm. [ 6.046945] systemd[1]: Detected architecture x86-64. [ 6.051637] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.064666] systemd[1]: No hostname configured. [ 6.069098] systemd[1]: Set hostname to . [ 6.123896] random: ln: uninitialized urandom read (6 bytes read) [ 6.210347] random: systemd: uninitialized urandom read (16 bytes read) [ 6.213288] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.217984] random: systemd: uninitialized urandom read (16 bytes read) [ 6.221201] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.226967] random: systemd: uninitialized urandom read (16 bytes read) [ 6.229631] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.233320] random: systemd: uninitialized urandom read (16 bytes read) [ 6.237042] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.243093] random: systemd: uninitialized urandom read (16 bytes read) [ 6.246604] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 6.253323] random: systemd: uninitialized urandom read (16 bytes read) [ 6.256614] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 6.262797] random: systemd: uninitialized urandom read (16 bytes read) [ 6.266113] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.272355] random: systemd: uninitialized urandom read (16 bytes read) [ 6.277560] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 6.296201] random: systemd: uninitialized urandom read (16 bytes read) [ 6.299147] systemd[1]: Reached target Sockets. [ OK ] Reached target Sockets. [ 6.310723] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 6.332157] systemd[1]: Starting Create Volatile Files and Directories... Starting Create Volatile Files and Directories... [ 6.357455] systemd[1]: Starting Journal Service... Starting Journal Service... [ 6.378887] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.423903] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 6.456432] systemd[1]: Starting Create Static Device Nodes in /dev... Starting Create Static Device Nodes in /dev... [ 6.508038] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 6.531446] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 6.556903] systemd[1]: Reached target Paths. [[ 6.567369] random: fast init done  OK ] Reached target Paths. [ 6.608864] tsc: Refined TSC clocksource calibration: 2399.982 MHz [ 6.617223] systemd[1]: memstrack.service: Succeeded. [ 6.622662] systemd[1]: systemd-vconsole-setup.service: Succeeded. [ 6.629016] systemd[1]: Started Setup Virtual Console. [ OK ] Started Setup Virtual Console. [ 6.642154] systemd[1]: Started Create Volatile Files and Directories. [ OK ] Started Create Volatile Files and Directories. [ 6.654301] systemd[1]: Started Create Static Device Nodes in /dev. [ 5.587165] [ OK ] Started Create Static Device Nodes in /dev. systemd-vconsole-setup[96]: KD_FONT_OP_GET failed while trying to get the font metadata: Invalid argument[ 6.676492] systemd[1]: Started Journal Service. [ OK ] Started Journal Service.[ 5.613567] systemd-vconsole-setup[ OK ] Started Apply Kernel Variables. [96]: Fonts will not be copied to remaining consoles [ 5.629774] systemd[1]: Started Apply Kernel Variables. [ 5.641453] Starting dracut ask for additional cmdline parameters...systemd[1]: Starting dracut ask for additional cmdline parameters... [ OK [ 5.697872] systemd] [1]: Started dracut ask for additional cmdline parameters. Started dracut ask for additional cmdline parameters. [ 5.711723] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 5.774807] dracut-cmdline[124]: dracut-8.10 (Green Obsidian) dracut-049-233.git20240115.el8 [ 5.789590] dracut-cmdline[124]: Using kernel command line parameters: ip=ens2:dhcp ifname=ens2:56:41:8d:5a:71:4a nameserver=192.168.200.253 rd.neednet kdumpnic=ens2 bootdev=ens2 rd.shell ro panic=1 nomodeset ipmtu=9000 rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr disable_cpu_apicid=0 acpi_rsdp=0xf52d0 elfcorehdr=3079540K [ 5.993888] dracut-cmdline[124]: Warning: Network interface 'ens2' does not exist [ OK ] Started dracut cmdline hook.[ 6.026660] systemd[1]: Started dracut cmdline hook. [ 6.031108] systemd[1]: Starting dracut pre-udev hook... Starting dracut pre-udev hook... [ OK ] Started dracut pre-udev hook. [ 6.176308] systemd[1]: Started dracut pre-udev hook. [ 6.183356] systemd[1]: Starting udev Kernel Device Manager... Starting udev Kernel Device Manager... [ OK [ 6.263603] systemd[1]: Started udev Kernel Device Manager.] Started udev Kernel Device Manager. [ 6.271373] systemd[1]: Starting udev Coldplug all Devices... Starting udev Coldplug all Devices... [ 6.447973] systemd[1]: Mounting Kernel Configuration File System... Mounting Kernel Configuration File System... [ 6.518670] systemd[1]: Mounted Kernel Configuration File System.[ OK ] Mounted Kernel Configuration File System. [ 6.597374] systemd[1]: Started udev Coldplug all Devices.[ OK ] Started udev Coldplug all Devices. [ 6.602723] systemd[ OK [1]: Reached target System Initialization. ] Reached target System Initialization. [ OK [ 6.613797] systemd[1]: Reached target Basic System.] Reached target Basic System. [[ 6.648036] systemd[1]: Started Hardware RNG Entropy Gatherer Daemon.  OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.672712] systemd[1]: Starting dracut initqueue hook... Starting dracut initqueue hook... [ 6.734745] rngd[245]: Disabling 7: PKCS11 Entropy generator (pkcs11) [ 6.743124] rngd[245]: Disabling 5: NIST Network Entropy Beacon (nist) [ 6.760191] rngd[245]: Disabling 9: Qrypt quantum entropy beacon (qrypt) [ 6.776186] rngd[245]: Initializing available sources [ 6.792393] rngd[245]: [hwrng ]: Initialization Failed [ 6.816975] rngd[245]: [rdrand]: Enabling RDRAND rng support [ 6.833270] [ 7.908029] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 rngd[245]: [rdrand]: Initialized [ 6.854591] rngd[245]: [jitter]: JITTER timeout set to 5 sec [ 7.940986] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 7.997528] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 8.041481] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 8.080660] scsi host0: ata_piix [ 7.014442] rngd[245]: [jitter]: Initializing AES buffer [ 8.089559] scsi host1: ata_piix [ 8.090568] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.093037] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.334926] systemd-udevd[242]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. [ 8.663297] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.7 MB/73.2 MiB) [ 8.684745] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.702882] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.726028] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.742711] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.763872] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.752454] systemd-udevd[243]: Using default interface naming scheme 'rhel-8.0'. [ 7.767909] systemd-udevd[243]: [ 8.845261] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=108 sclass=31 link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. [ 8.272804] systemd-udevd[242]: Using default interface naming scheme 'rhel-8.0'. [ 9.246290] dracut-initqueue[286]: Starting dhcp for interface ens2 [ 9.499595] dracut-initqueue[318]: dhcp: PREINIT ens2 up [ 9.647205] dhclient[317]: Created duid "\000\004\250,\\\342\272\373Bu\271t\314}\203\3007B". [ 9.666581] dhclient[317]: DHCPDISCOVER on ens2 to 255.255.255.255 port 67 interval 4 (xid=0x85a28e40) [ 9.682554] dhclient[317]: DHCPREQUEST on ens2 to 255.255.255.255 port 67 (xid=0x85a28e40) [ 9.696632] dhclient[317]: DHCPOFFER from 192.168.203.254 [ 9.706569] dhclient[317]: DHCPACK from 192.168.203.254 (xid=0x85a28e40) [ 9.838680] dracut-initqueue[336]: dhcp: BOND setting ens2 [ 11.432082] rngd[245]: [jitter]: Unable to obtain AES key, disabling JITTER source [ 12.532245] random: crng init done [ 11.463096] rngd[245]: [jitter]: Initialization Failed [ 11.900683] dhclient[317]: bound to 192.168.203.146 -- renewal in 27945 seconds. [ OK ] Started dracut initqueue hook. [ 12.640887] systemd[1]: Started dracut initqueue hook. [[ 12.648805] systemd OK ] Reached target Remote File Systems (Pre). [ OK [1]: Reached target Remote File Systems (Pre).] Reached target Remote File Systems. [ 12.665336] systemd [[1]: Reached target Remote File Systems. OK ] Reached target Initrd Root File System. [ [ 12.673595]  OK ] Reached target Initrd File Systems.systemd[1]: Reached target Initrd Root File System. [ OK ] Reached target Initrd Default Target.[ 12.686798] systemd[1]: Reached target Initrd File Systems. [ 12.696301] systemd[1]: Starting dracut pre-pivot and cleanup hook... Reached target Initrd Default Target. [ 12.709551] systemd[1]: Starting dracut pre-pivot and cleanup hook... [ OK [ 12.920557] systemd[1]: Started dracut pre-pivot and cleanup hook. ] Started dracut pre-pivot and cleanup hook. [ 12.930680] systemd[1]: Starting Kdump Vmcore Save Service... Starting Kdump Vmcore Save Service... [ 13.067755] kdump[410]: Kdump is using the default log level(3). [ 13.156272] kdump[435]: saving to crashdump@192.168.200.253:/tmp/kcrashes/oleg346-server//192.168.203.146-2026-09-17-19:18:47 [ 13.853110] kdump[439]: saving vmcore-dmesg.txt to crashdump@192.168.200.253:/tmp/kcrashes/oleg346-server//192.168.203.146-2026-09-17-19:18:47 [ 14.094592] kdump.sh[441]: 328+0 records in [ 14.105422] kdump.sh[441]: 328+0 records out [ 14.126532] kdump.sh[441]: 167936 bytes (168 kB, 164 KiB) copied, 0.000307897 s, 545 MB/s [ 14.354903] kdump[444]: saving vmcore-dmesg.txt complete [ 14.373348] kdump[446]: saving vmcore [ 21.033223] kdump.sh[449]: Checking for memory holes : [ 0.0 %] / Checking for memory holes : [100.0 %] | Excluding unnecessary pages : [ 0.0 %] \ Excluding unnecessary pages : [100.0 %] - Copying data : [ 26.1 %] / eta: 2s Copying data : [ 33.3 %] | eta: 4s Copying data : [ 37.7 %] \ eta: 5s Copying data : [ 51.3 %] - eta: 4s Copying data : [ 71.8 %] / eta: 2s Copying data : [ 92.5 %] | eta: 1s Copying data : [100.0 %] \ eta: 0s Copying data : [100.0 %] - eta: 0s [ 21.094131] kdump.sh[449]: The dumpfile is saved to STDOUT. [ 21.100182] kdump.sh[449]: makedumpfile Completed. [ 21.108251] kdump.sh[450]: 329233+409 records in [ 21.116156] kdump.sh[450]: 329233+409 records out [ 21.121400] kdump.sh[450]: 168645885 bytes (169 MB, 161 MiB) copied, 6.40132 s, 26.3 MB/s [ 21.368442] kdump[453]: saving vmcore complete [ 21.400322] kdump[455]: saving the /run/initramfs/kexec-dmesg.log to crashdump@192.168.200.253:/tmp/kcrashes/oleg346-server//192.168.203.146-2026-09-17-19:18:47/ [ 21.682247] kdump[462]: Executing final action systemctl reboot -f [ 21.710326] systemd[1]: Shutting down. [ 22.824611] systemd-shutdown[1]: Syncing filesystems and block devices. [ 22.829270] systemd-shutdown[1]: Sending SIGTERM to remaining processes... [ 22.841196] systemd-shutdown[1]: Sending SIGKILL to remaining processes... [ 22.847067] systemd-shutdown[1]: Unmounting file systems. [ 22.850086] [466]: Remounting '/' read-only in with options 'size=106908k,nr_inodes=26727'. [ 22.854415] systemd-shutdown[1]: All filesystems unmounted. [ 22.856241] systemd-shutdown[1]: Deactivating swaps. [ 22.857872] systemd-shutdown[1]: All swaps deactivated. [ 22.861231] systemd-shutdown[1]: Detaching loop devices. [ 22.863445] systemd-shutdown[1]: All loop devices detached. [ 22.865491] systemd-shutdown[1]: Stopping MD devices. [ 22.867395] systemd-shutdown[1]: All MD devices stopped. [ 22.869452] systemd-shutdown[1]: Detaching DM devices. [ 22.871678] systemd-shutdown[1]: All DM devices detached. [ 22.873815] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached. [ 22.878243] systemd-shutdown[1]: Syncing filesystems and block devices. [ 22.881739] systemd-shutdown[1]: Rebooting. [ 22.883393] Unregister pv shared memory for cpu 0 [ 22.885790] Restarting system. [ 22.886979] reboot: machine restart