[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483765300 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002393] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009022] pid_max: default: 32768 minimum: 301 [ 0.010173] LSM: Security Framework initializing [ 0.011089] Yama: becoming mindful. [ 0.012050] SELinux: Initializing. [ 0.013117] *** VALIDATE selinux *** [ 0.021702] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026904] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029136] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030103] *** VALIDATE tmpfs *** [ 0.031418] *** VALIDATE proc *** [ 0.032183] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035171] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038037] Spectre V2 : User space: Vulnerable [ 0.039015] Speculative Store Bypass: Vulnerable [ 0.042103] debug: unmapping init [mem 0xffffffff8ec59000-0xffffffff8ec60fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047017] ... bit width: 48 [ 0.048020] ... generic registers: 4 [ 0.049018] ... value mask: 0000ffffffffffff [ 0.050020] ... max period: 00007fffffffffff [ 0.051020] ... fixed-purpose events: 3 [ 0.052021] ... event mask: 000000070000000f [ 0.053383] rcu: Hierarchical SRCU implementation. [ 0.055512] smp: Bringing up secondary CPUs ... [ 0.056668] x86: Booting SMP configuration: [ 0.057039] .... node #0, CPUs: #1 #2 #3 [ 0.060375] smp: Brought up 1 node, 4 CPUs [ 0.062021] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.242316] node 0 deferred pages initialised in 177ms [ 0.245205] devtmpfs: initialized [ 0.246250] x86/mm: Memory block size: 128MB [ 0.248908] gcov: version magic: 0x41383552 [ 0.250376] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.251101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.252285] pinctrl core: initialized pinctrl subsystem [ 0.253276] [ 0.253921] ************************************************************* [ 0.254018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.255016] ** ** [ 0.256018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.257025] ** ** [ 0.258014] ** This means that this kernel is built to expose internal ** [ 0.259020] ** IOMMU data structures, which may compromise security on ** [ 0.260015] ** your system. ** [ 0.261023] ** ** [ 0.262018] ** If you see this message and you are not debugging the ** [ 0.263021] ** kernel, report this immediately to your vendor! ** [ 0.264016] ** ** [ 0.265018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.266028] ************************************************************* [ 0.267653] NET: Registered protocol family 16 [ 0.268502] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.269096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.270086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.271520] cpuidle: using governor menu [ 0.272880] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.275644] PCI: Using configuration type 1 for base access [ 0.277134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.289066] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.290023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.293033] cryptd: max_cpu_qlen set to 1000 [ 0.297353] ACPI: Added _OSI(Module Device) [ 0.298016] ACPI: Added _OSI(Processor Device) [ 0.299013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.299921] ACPI: Added _OSI(Processor Aggregator Device) [ 0.303232] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.306531] ACPI: Interpreter enabled [ 0.307080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.308017] ACPI: Using IOAPIC for interrupt routing [ 0.309145] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.310452] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.320638] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.321049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.322028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.323089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.326142] acpiphp: Slot [2] registered [ 0.327000] acpiphp: Slot [5] registered [ 0.329229] acpiphp: Slot [6] registered [ 0.331247] acpiphp: Slot [7] registered [ 0.333236] acpiphp: Slot [8] registered [ 0.335180] acpiphp: Slot [9] registered [ 0.336140] acpiphp: Slot [10] registered [ 0.338183] acpiphp: Slot [3] registered [ 0.340138] acpiphp: Slot [4] registered [ 0.341217] acpiphp: Slot [11] registered [ 0.342142] acpiphp: Slot [12] registered [ 0.344145] acpiphp: Slot [13] registered [ 0.345185] acpiphp: Slot [14] registered [ 0.347166] acpiphp: Slot [15] registered [ 0.348153] acpiphp: Slot [16] registered [ 0.350203] acpiphp: Slot [17] registered [ 0.352144] acpiphp: Slot [18] registered [ 0.353156] acpiphp: Slot [19] registered [ 0.355145] acpiphp: Slot [20] registered [ 0.356135] acpiphp: Slot [21] registered [ 0.358130] acpiphp: Slot [22] registered [ 0.359153] acpiphp: Slot [23] registered [ 0.361138] acpiphp: Slot [24] registered [ 0.363193] acpiphp: Slot [25] registered [ 0.364148] acpiphp: Slot [26] registered [ 0.366139] acpiphp: Slot [27] registered [ 0.367185] acpiphp: Slot [28] registered [ 0.369148] acpiphp: Slot [29] registered [ 0.371153] acpiphp: Slot [30] registered [ 0.373131] acpiphp: Slot [31] registered [ 0.374130] PCI host bridge to bus 0000:00 [ 0.376032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.378029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.381035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.383032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.386028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.389026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.391195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.395138] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.398363] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.408825] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.414000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.416022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.419027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.421022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.424614] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.427829] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.430065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.433960] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.439024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.452024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.457916] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.463244] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.537025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.547031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.572026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.585079] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.596023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.604020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.620026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.629990] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.640024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.648026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.667022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.679165] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.687016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.697023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.710025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.720000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.727019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.735020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.753017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.765690] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.772020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.777015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.791018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.802604] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.805429] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.808383] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.811417] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.814338] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.819030] iommu: Default domain type: Passthrough [ 0.821439] SCSI subsystem initialized [ 0.823161] ACPI: bus type USB registered [ 0.825125] usbcore: registered new interface driver usbfs [ 0.827110] usbcore: registered new interface driver hub [ 0.829127] usbcore: registered new device driver usb [ 0.830195] pps_core: LinuxPPS API ver. 1 registered [ 0.833013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.836063] PTP clock support registered [ 0.838186] EDAC MC: Ver: 3.0.0 [ 0.840121] PCI: Using ACPI for IRQ routing [ 0.841848] NetLabel: Initializing [ 0.843015] NetLabel: domain hash size = 128 [ 0.845020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.847123] NetLabel: unlabeled traffic allowed by default [ 0.851229] vgaarb: loaded [ 0.853157] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.855016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.863000] clocksource: Switched to clocksource kvm-clock [ 0.979224] VFS: Disk quotas dquot_6.6.0 [ 0.981043] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.983885] *** VALIDATE ramfs *** [ 0.985423] *** VALIDATE hugetlbfs *** [ 0.987284] pnp: PnP ACPI init [ 0.989852] pnp: PnP ACPI: found 6 devices [ 1.007911] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.011847] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.014468] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.016890] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.019599] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.022208] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.025301] NET: Registered protocol family 2 [ 1.027741] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.032504] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.035877] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.041665] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.045519] TCP: Hash tables configured (established 65536 bind 65536) [ 1.048423] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.051651] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.054347] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.056798] NET: Registered protocol family 1 [ 1.059637] RPC: Registered named UNIX socket transport module. [ 1.062439] RPC: Registered udp transport module. [ 1.064589] RPC: Registered tcp transport module. [ 1.066576] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.069284] NET: Registered protocol family 44 [ 1.070978] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.073470] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.075674] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.078395] PCI: CLS 0 bytes, default 64 [ 1.080804] Unpacking initramfs... [ 2.544233] debug: unmapping init [mem 0xffff8dba7cc54000-0xffff8dba7ffbffff] [ 2.548762] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.551722] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.555248] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.089583] Initialise system trusted keyrings [ 3.091600] Key type blacklist registered [ 3.094060] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.104834] zbud: loaded [ 3.109138] *** VALIDATE nfs *** [ 3.110722] *** VALIDATE nfs4 *** [ 3.112842] pstore: using deflate compression [ 3.118444] Platform Keyring initialized [ 3.255431] NET: Registered protocol family 38 [ 3.256870] Key type asymmetric registered [ 3.257991] Asymmetric key parser 'x509' registered [ 3.259734] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.263532] io scheduler mq-deadline registered [ 3.265829] io scheduler kyber registered [ 3.267942] io scheduler bfq registered [ 3.270876] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.274907] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.278793] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.282108] ACPI: Power Button [PWRF] [ 3.289694] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.306616] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.327403] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.337595] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.351271] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.378232] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.406799] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.414222] Non-volatile memory driver v1.3 [ 3.416476] Linux agpgart interface v0.103 [ 3.458873] virtio_blk virtio1: [vda] 67992 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.462161] vda: detected capacity change from 0 to 34811904 [ 3.479239] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.482541] vdb: detected capacity change from 0 to 1073741824 [ 3.506287] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.508587] vdc: detected capacity change from 0 to 2621440000 [ 3.522942] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.525933] vdd: detected capacity change from 0 to 2621440000 [ 3.540559] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.543823] vde: detected capacity change from 0 to 4294967296 [ 3.559366] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.562756] vdf: detected capacity change from 0 to 4294967296 [ 3.569406] libphy: Fixed MDIO Bus: probed [ 3.580221] usbcore: registered new interface driver usbserial_generic [ 3.582837] usbserial: USB Serial support registered for generic [ 3.586219] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.591552] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.594070] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.597519] mousedev: PS/2 mouse device common for all mice [ 3.599958] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.602632] rtc_cmos 00:05: RTC can wake from S4 [ 3.608503] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.608882] rtc_cmos 00:05: registered as rtc0 [ 3.615981] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.616497] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.618834] intel_pstate: CPU model not supported [ 3.625199] hid: raw HID events driver (C) Jiri Kosina [ 3.626884] usbcore: registered new interface driver usbhid [ 3.628377] usbhid: USB HID core driver [ 3.629544] drop_monitor: Initializing network drop monitor service [ 3.631773] Initializing XFRM netlink socket [ 3.633818] NET: Registered protocol family 10 [ 3.636554] Segment Routing with IPv6 [ 3.637801] NET: Registered protocol family 17 [ 3.639516] mpls_gso: MPLS GSO support [ 3.645503] RAS: Correctable Errors collector initialized. [ 3.647341] AVX version of gcm_enc/dec engaged. [ 3.648606] AES CTR mode by8 optimization enabled [ 3.744514] sched_clock: Marking stable (3744473778, 0)->(4698568574, -954094796) [ 3.751360] registered taskstats version 1 [ 3.753757] Loading compiled-in X.509 certificates [ 3.760245] zswap: loaded using pool lzo/zbud [ 3.787459] Key type big_key registered [ 3.800061] Key type encrypted registered [ 3.800954] ima: No TPM chip found, activating TPM-bypass! [ 3.802635] ima: Allocated hash algorithm: sha1 [ 3.803932] ima: No architecture policies found [ 3.805558] evm: Initialising EVM extended attributes: [ 3.807064] evm: security.selinux [ 3.807941] evm: security.ima [ 3.808951] evm: security.capability [ 3.809962] evm: HMAC attrs: 0x1 [ 3.811671] rtc_cmos 00:05: setting system clock to 2026-04-14 20:28:19 UTC (1776198499) [ 3.817496] debug: unmapping init [mem 0xffffffff8fc03000-0xffffffff8fdfffff] [ 3.819851] debug: unmapping init [mem 0xffffffff8e982000-0xffffffff8ec58fff] [ 3.828070] Write protecting the kernel read-only data: 28672k [ 3.830555] debug: unmapping init [mem 0xffffffff8d003000-0xffffffff8d1fffff] [ 3.832856] debug: unmapping init [mem 0xffffffff8d914000-0xffffffff8d9fffff] [ 3.862754] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.869654] systemd[1]: Detected virtualization kvm. [ 3.871717] systemd[1]: Detected architecture x86-64. [ 3.873817] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.898912] systemd[1]: No hostname configured. [ 3.900343] systemd[1]: Set hostname to . [ 3.901900] random: systemd: uninitialized urandom read (16 bytes read) [ 3.903756] systemd[1]: Initializing machine ID from random generator. [ 4.039374] random: systemd: uninitialized urandom read (16 bytes read) [ 4.042379] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.046737] random: systemd: uninitialized urandom read (16 bytes read) [ 4.050501] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.054317] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.690563] device-mapper: uevent: version 1.0.3 [ 4.693391] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook...[ 5.046852] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.538661] virtio_net virtio0 ens2: renamed from eth0 [ 5.590451] scsi host0: ata_piix [ 5.614346] scsi host1: ata_piix [ 5.616051] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.618684] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.299557] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.314525] random: crng init done [ 10.315617] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.708846] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.739687] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.016325] SELinux: Disabled at runtime. [ 12.075661] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.083989] systemd[1]: Detected virtualization kvm. [ 12.085637] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.533254] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.535881] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.539652] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.542969] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.545255] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.551866] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.556819] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ 12.611132] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.963877] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.250268] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.268244] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.400817] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.417726] EDAC sbridge: Ver: 1.1.2 [ 15.101033] Key type dns_resolver registered [ 15.401503] NFS: Registering the id_resolver key type [ 15.403735] Key type id_resolver registered [ 15.405331] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg230-server login: [ 39.470654] spl: loading out-of-tree module taints kernel. [ 45.250171] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 52.202336] hrtimer: interrupt took 2701397 ns [ 57.588921] alg: No test for adler32 (adler32-zlib) [ 58.345547] Key type ._llcrypt registered [ 58.348826] Key type .llcrypt registered [ 58.481465] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 76.984953] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 78.728201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 79.272166] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 80.102981] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 80.108928] LNet: Accept secure, port 988 [ 81.879120] Key type lgssc registered [ 83.769081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.438861] vdc: vdc1 vdc9 [ 103.854635] vde: vde1 vde9 [ 113.207089] vdf: vdf1 vdf9 [ 129.313579] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 138.030592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 138.305686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 138.406636] Lustre: lustre-MDT0000: new disk, initializing [ 138.813899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.881972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.606606] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 152.796462] Lustre: lustre-OST0000: new disk, initializing [ 152.803740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 152.807850] Lustre: Skipped 1 previous similar message [ 152.888777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.431401] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 166.609754] Lustre: lustre-OST0001: new disk, initializing [ 166.612375] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 166.681849] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.230399] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 183.273194] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 190.669807] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 201.268748] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing check_logdir /tmp/testlogs/ [ 206.270233] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing yml_node [ 211.534743] Lustre: DEBUG MARKER: Client: 2.15.8.2 [ 213.909321] Lustre: DEBUG MARKER: MDS: 2.15.8.2 [ 216.704459] Lustre: DEBUG MARKER: OSS: 2.15.8.2 [ 218.527530] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Tue Apr 14 16:31:52 EDT 2026 [ 226.156975] Lustre: DEBUG MARKER: excepting tests: 18b 23b [ 234.046773] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 244.068417] Lustre: server umount lustre-MDT0000 complete [ 248.881857] LustreError: 7841:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198744 with bad export cookie 14469687902118320453 [ 248.897350] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 251.807179] Lustre: 3019:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198740/real 1776198740] req@000000005d064de6 x1862479197000384/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776198747 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 251.817594] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.943213] Lustre: 10019:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198744/real 1776198744] req@0000000026529568 x1862479197000448/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776198750 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 255.039100] Lustre: server umount lustre-OST0000 complete [ 256.994126] Lustre: 3017:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198745/real 1776198745] req@00000000f6665d01 x1862479197000640/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776198752 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 259.132933] Lustre: server umount lustre-OST0001 complete [ 269.924950] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 272.273776] Key type lgssc unregistered [ 272.459680] LNet: 10822:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 272.469865] LNet: Removed LNI 192.168.202.130@tcp [ 273.042576] Key type .llcrypt unregistered [ 273.046450] Key type ._llcrypt unregistered [ 287.718289] alg: No test for adler32 (adler32-zlib) [ 288.474778] Key type ._llcrypt registered [ 288.479176] Key type .llcrypt registered [ 288.706954] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 301.595331] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 302.732784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 303.354389] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 303.516730] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 303.519822] LNet: Accept secure, port 988 [ 305.199169] Key type lgssc registered [ 306.557929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 315.025298] vdc: vdc1 vdc9 [ 315.057686] vdc: vdc1 vdc9 [ 315.073791] vdc: vdc1 vdc9 [ 323.356548] vde: vde1 vde9 [ 323.393566] vde: vde1 vde9 [ 334.982062] vdf: vdf1 vdf9 [ 356.879871] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 368.461447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 368.651427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 368.743875] Lustre: lustre-MDT0000: new disk, initializing [ 369.285139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 369.379054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 375.164343] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 386.851508] Lustre: lustre-OST0000: new disk, initializing [ 386.859995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 386.869029] Lustre: Skipped 1 previous similar message [ 386.974400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.955262] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 401.545348] Lustre: lustre-OST0001: new disk, initializing [ 401.550392] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 401.690928] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 405.868745] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 415.775665] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 424.342721] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 431.659929] Lustre: DEBUG MARKER: == sanity-lfsck test 0: Control LFSCK manually =========== 16:35:26 (1776198926) [ 439.488916] LustreError: 17315:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 sleeping for 3000ms [ 442.575912] LustreError: 17315:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 awake [ 444.203034] LustreError: 17560:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 sleeping for 3000ms [ 446.223081] LustreError: 17560:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 455.648197] Lustre: server umount lustre-MDT0000 complete [ 459.251801] LustreError: 14690:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198954 with bad export cookie 5102148536753772684 [ 459.260132] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 464.355528] Lustre: 18266:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198954/real 1776198954] req@00000000e86bd859 x1862479431890688/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776198960 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 464.485124] Lustre: server umount lustre-OST0000 complete [ 466.912916] Lustre: 12165:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198955/real 1776198955] req@00000000cc619516 x1862479431890880/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776198962 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 466.937219] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.971682] Lustre: server umount lustre-OST0001 complete [ 475.485746] Lustre: DEBUG MARKER: == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 16:36:10 (1776198970) [ 491.397276] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 502.109413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.858309] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 514.135962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 518.184825] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.221833] Lustre: lustre-OST0000: deleting orphan objects from 0x0:43 to 0x0:65 [ 519.180434] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 523.232482] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.942976] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 528.955031] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:65 [ 531.595290] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 539.057530] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 542.480771] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 556.695628] Lustre: *** cfs_fail_loc=1501, val=0*** [ 571.813526] Lustre: DEBUG MARKER: == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 16:37:46 (1776199066) [ 573.166773] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_1b OI Scrub not implemented for ZFS [ 574.761847] Lustre: DEBUG MARKER: == sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 16:37:49 (1776199069) [ 581.988789] Lustre: *** cfs_fail_loc=1504, val=0*** [ 581.993503] Lustre: *** cfs_fail_loc=1504, val=0*** [ 597.901713] Lustre: DEBUG MARKER: == sanity-lfsck test 1d: LFSCK can fix mismatch of FID in LMA and FID in child linkea ========================================================== 16:38:12 (1776199092) [ 599.548687] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_1d needs >= 2 MDTs [ 601.346875] Lustre: DEBUG MARKER: == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 16:38:15 (1776199095) [ 610.150607] Lustre: *** cfs_fail_loc=1603, val=0*** [ 627.926985] Lustre: DEBUG MARKER: == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 16:38:41 (1776199121) [ 636.983093] Lustre: *** cfs_fail_loc=1604, val=0*** [ 650.855254] Lustre: DEBUG MARKER: == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 16:39:05 (1776199145) [ 659.099795] Lustre: *** cfs_fail_loc=1605, val=0*** [ 672.518103] Lustre: DEBUG MARKER: == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 16:39:26 (1776199166) [ 680.724942] Lustre: *** cfs_fail_loc=161d, val=0*** [ 695.466097] Lustre: DEBUG MARKER: == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 16:39:49 (1776199189) [ 696.842407] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_2e needs >= 2 MDTs [ 698.267910] Lustre: DEBUG MARKER: == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 16:39:52 (1776199192) [ 706.944570] Lustre: *** cfs_fail_loc=1603, val=0*** [ 708.005399] Lustre: *** cfs_fail_loc=1604, val=0*** [ 718.172941] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 16:40:12 (1776199212) [ 719.902169] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_4 OI Scrub not implemented for ZFS [ 721.592509] Lustre: DEBUG MARKER: == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 16:40:15 (1776199215) [ 722.885635] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_5 OI Scrub not implemented for ZFS [ 724.457360] Lustre: DEBUG MARKER: == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 16:40:18 (1776199218) [ 734.056290] LustreError: 26048:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 sleeping for 1000ms [ 735.111131] LustreError: 26048:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 awake [ 736.175745] LustreError: 26048:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 awake [ 736.193893] LustreError: 26048:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 sleeping for 1000ms [ 736.207336] LustreError: 26048:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 738.312322] LustreError: 26048:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 awake [ 738.319234] LustreError: 26048:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 739.405643] Lustre: *** cfs_fail_loc=1608, val=0*** [ 742.576499] LustreError: 26347:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 sleeping for 1000ms [ 742.592386] LustreError: 26347:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 743.639145] LustreError: 26347:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 awake [ 743.641972] LustreError: 26347:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 745.576191] LustreError: 26347:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 752.949315] Lustre: DEBUG MARKER: == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 16:40:47 (1776199247) [ 763.196493] LustreError: 26842:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 sleeping for 1000ms [ 763.204100] LustreError: 26842:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 764.272707] LustreError: 26842:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 awake [ 764.288525] LustreError: 26842:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 770.567262] Lustre: *** cfs_fail_loc=1609, val=0*** [ 779.634309] LustreError: 27189:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 sleeping for 1000ms [ 779.643931] LustreError: 27189:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 11 previous similar messages [ 779.752450] LustreError: 27189:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 786.887923] Lustre: DEBUG MARKER: == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 16:41:21 (1776199281) [ 799.531892] LustreError: 27790:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 awake [ 799.539841] LustreError: 27790:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 11 previous similar messages [ 804.061692] Lustre: Failing over lustre-MDT0000 [ 805.187948] Lustre: server umount lustre-MDT0000 complete [ 813.149044] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 813.175484] Lustre: Evicted from MGS (at 192.168.202.130@tcp) after server handle changed from 0x46ce790fd47d005c to 0x46ce790fd47f5928 [ 813.197868] Lustre: MGC192.168.202.130@tcp: Connection restored to 192.168.202.130@tcp (at 0@lo) [ 813.802619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 814.114349] Lustre: lustre-OST0000: deleting orphan objects from 0x0:485 to 0x0:513 [ 814.121907] Lustre: lustre-OST0001: deleting orphan objects from 0x0:485 to 0x0:513 [ 819.180552] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.203647] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 819.396732] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 828.504541] Lustre: DEBUG MARKER: == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 16:42:02 (1776199322) [ 841.082404] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 852.959372] Lustre: 12166:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199304/real 1776199304] req@000000009e884001 x1862479432125824/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776199348 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 852.985622] Lustre: 12166:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 856.543716] Lustre: 12168:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199308/real 1776199308] req@000000004e2bf0a7 x1862479432125952/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776199352 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 860.537836] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 863.734663] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 878.066963] Lustre: *** cfs_fail_loc=1604, val=0*** [ 880.930645] LustreError: 30812:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 sleeping for 1000ms [ 880.936243] LustreError: 30812:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 881.999123] LustreError: 30812:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 awake [ 882.003107] LustreError: 30812:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 5 previous similar messages [ 885.415451] Lustre: Failing over lustre-MDT0000 [ 885.731579] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.742164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 885.763177] Lustre: Skipped 2 previous similar messages [ 885.775391] Lustre: Skipped 1 previous similar message [ 886.536244] Lustre: server umount lustre-MDT0000 complete [ 894.149515] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 894.169606] Lustre: Evicted from MGS (at 192.168.202.130@tcp) after server handle changed from 0x46ce790fd47f5928 to 0x46ce790fd47fa525 [ 894.183592] Lustre: MGC192.168.202.130@tcp: Connection restored to (at 0@lo) [ 894.195963] Lustre: Skipped 1 previous similar message [ 894.586330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 894.717351] Lustre: lustre-OST0000: deleting orphan objects from 0x0:566 to 0x0:609 [ 894.721219] Lustre: lustre-OST0001: deleting orphan objects from 0x0:565 to 0x0:609 [ 894.864314] LustreError: 31487:0:(osd_index.c:1884:osd_index_it_get()) NOT IMPLEMETED YET (move to 0xa113000002000000) [ 899.127121] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 901.666253] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 907.968301] Lustre: DEBUG MARKER: == sanity-lfsck test 8: LFSCK state machine ============== 16:43:22 (1776199402) [ 909.990984] Lustre: server umount lustre-MDT0000 complete [ 914.120881] LustreError: 19758:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199409 with bad export cookie 5102148536753956133 [ 914.128368] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 919.199161] Lustre: 12167:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199407/real 1776199407] req@00000000d55d0692 x1862479432165568/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776199414 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 919.236066] Lustre: 12167:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 919.246468] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.101680] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 938.386039] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 947.017697] vdc: vdc1 vdc9 [ 955.295440] vde: vde1 vde9 [ 955.357175] vde: vde1 vde9 [ 955.382899] vde: vde1 vde9 [ 966.031386] vdf: vdf1 vdf9 [ 985.122438] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 992.734796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 992.805380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 992.928270] Lustre: lustre-MDT0000: new disk, initializing [ 993.353162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.402418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 997.902151] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1005.694991] Lustre: lustre-OST0000: new disk, initializing [ 1005.697946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1005.701295] Lustre: Skipped 1 previous similar message [ 1009.987417] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1017.819904] Lustre: lustre-OST0001: new disk, initializing [ 1017.823895] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1021.687741] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1030.873127] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1034.577436] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1051.733774] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1052.571490] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1052.573377] Lustre: Skipped 19 previous similar messages [ 1056.107712] LustreError: 38327:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 sleeping for 2000ms [ 1056.110147] LustreError: 38327:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 4 previous similar messages [ 1058.111196] LustreError: 38327:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 awake [ 1058.116881] LustreError: 38327:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 4 previous similar messages [ 1061.752778] Lustre: *** cfs_fail_loc=1609, val=0*** [ 1067.047144] Lustre: *** cfs_fail_loc=160a, val=0*** [ 1073.554430] Lustre: Failing over lustre-MDT0000 [ 1073.788376] Lustre: server umount lustre-MDT0000 complete [ 1073.790882] Lustre: Skipped 2 previous similar messages [ 1080.349752] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1080.363086] Lustre: Evicted from MGS (at 192.168.202.130@tcp) after server handle changed from 0x46ce790fd47fa9a8 to 0x46ce790fd47febdb [ 1080.370696] Lustre: MGC192.168.202.130@tcp: Connection restored to (at 0@lo) [ 1080.373521] Lustre: Skipped 1 previous similar message [ 1080.933806] Lustre: *** cfs_fail_loc=160b, val=0*** [ 1080.948501] Lustre: lustre-OST0000: deleting orphan objects from 0x0:46 to 0x0:65 [ 1080.950050] Lustre: lustre-OST0001: deleting orphan objects from 0x0:46 to 0x0:65 [ 1081.887231] Lustre: 12167:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199570/real 1776199570] req@00000000ce9c518b x1862479432193344/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776199577 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 1081.907638] Lustre: 12167:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1081.912813] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.356665] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1091.061985] Lustre: Failing over lustre-MDT0000 [ 1092.064636] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.083904] Lustre: Skipped 1 previous similar message [ 1092.102239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1092.113103] Lustre: Skipped 1 previous similar message [ 1099.201475] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1099.210529] Lustre: Evicted from MGS (at 192.168.202.130@tcp) after server handle changed from 0x46ce790fd47febdb to 0x46ce790fd47fee89 [ 1099.220455] Lustre: MGC192.168.202.130@tcp: Connection restored to (at 0@lo) [ 1099.224835] Lustre: Skipped 2 previous similar messages [ 1099.603546] Lustre: *** cfs_fail_loc=160b, val=0*** [ 1099.614798] Lustre: lustre-OST0001: deleting orphan objects from 0x0:46 to 0x0:97 [ 1099.649781] Lustre: lustre-OST0000: deleting orphan objects from 0x0:46 to 0x0:97 [ 1103.116879] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1107.712092] Lustre: Failing over lustre-MDT0000 [ 1108.015193] Lustre: server umount lustre-MDT0000 complete [ 1108.017693] Lustre: Skipped 1 previous similar message [ 1113.186980] Lustre: Evicted from MGS (at 192.168.202.130@tcp) after server handle changed from 0x46ce790fd47fee89 to 0x46ce790fd47ff1df [ 1113.524144] Lustre: lustre-OST0001: deleting orphan objects from 0x0:46 to 0x0:129 [ 1113.524464] Lustre: lustre-OST0000: deleting orphan objects from 0x0:46 to 0x0:129 [ 1116.978730] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1118.694896] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.706038] Lustre: Skipped 1 previous similar message [ 1118.721838] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1118.730399] Lustre: Skipped 3 previous similar messages [ 1119.712753] Lustre: 12167:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199608/real 1776199608] req@000000003825264c x1862479432204352/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776199615 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1119.742719] Lustre: 12167:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1126.503088] LustreError: 42473:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 1133.933159] Lustre: DEBUG MARKER: == sanity-lfsck test 9a: LFSCK speed control (1) ========= 16:47:08 (1776199628) [ 1147.414326] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1166.871372] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1169.992523] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1291.565172] Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) ========= 16:49:46 (1776199786) [ 1345.164197] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1345.171168] Lustre: Skipped 4 previous similar messages [ 1369.677722] Lustre: *** cfs_fail_loc=160c, val=0*** [ 1369.680338] Lustre: Skipped 1 previous similar message [ 1411.267559] Lustre: DEBUG MARKER: == sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 16:51:45 (1776199905) [ 1413.356864] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_10 lookup(..)/linkea on ZFS issue [ 1415.167640] Lustre: DEBUG MARKER: == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 16:51:49 (1776199909) [ 1464.299763] Lustre: server umount lustre-MDT0000 complete [ 1469.239552] LustreError: 46480:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199964 with bad export cookie 5102148536753975775 [ 1469.262900] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1469.271798] LustreError: Skipped 1 previous similar message [ 1474.530536] Lustre: 46801:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776199964/real 1776199964] req@00000000d9eb747d x1862479433176896/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776199970 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1474.575282] Lustre: 46801:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1519.393474] LustreError: 48077:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 1: rc = -5 [ 1530.656558] LustreError: 48077:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 1559.391189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.409910] Lustre: Skipped 5 previous similar messages [ 1563.119812] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1566.309950] Lustre: 48617:0:(ofd_dev.c:577:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 1566.322705] Lustre: *** cfs_fail_loc=160e, val=0*** [ 1569.385187] Lustre: 48617:0:(ofd_dev.c:589:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 1578.599661] Lustre: DEBUG MARKER: == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 16:54:32 (1776200072) [ 1593.157955] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1602.895086] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1603.166146] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2736 to 0x0:2753 [ 1606.955738] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1608.160208] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1613.281122] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1619.032326] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2671 to 0x0:2689 [ 1621.482663] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1628.393683] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1631.159554] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1640.234995] Lustre: *** cfs_fail_loc=160d, val=0*** [ 1645.759623] Lustre: Failing over lustre-OST0000 [ 1645.849531] Lustre: server umount lustre-OST0000 complete [ 1645.853398] Lustre: Skipped 2 previous similar messages [ 1649.119866] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1649.126430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.140359] Lustre: Skipped 1 previous similar message [ 1649.147821] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.691092] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1654.139713] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1654.245714] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [ 1654.245863] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1654.252702] Lustre: Skipped 1 previous similar message [ 1654.259416] Lustre: *** cfs_fail_loc=215, val=0*** [ 1654.282011] Lustre: Skipped 1 previous similar message [ 1656.664217] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1659.363492] Lustre: *** cfs_fail_loc=215, val=0*** [ 1659.853103] Lustre: 52995:0:(ofd_dev.c:577:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 1659.877744] Lustre: 52995:0:(ofd_dev.c:589:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 1662.427089] Lustre: Failing over lustre-OST0000 [ 1662.946620] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1668.941753] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1670.223271] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1670.320490] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1670.330161] Lustre: *** cfs_fail_loc=215, val=0*** [ 1670.336583] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2818 to 0x0:2849 [ 1673.374852] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1675.744635] Lustre: *** cfs_fail_loc=215, val=0*** [ 1682.894212] LustreError: 48081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200178 with bad export cookie 5102148536754580169 [ 1682.900978] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1686.431163] Lustre: 12168:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200175/real 1776200175] req@0000000037565e6e x1862479433199616/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200182 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 1686.448072] Lustre: 12168:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1699.874300] Lustre: DEBUG MARKER: == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 16:56:34 (1776200194) [ 1701.090238] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_12a needs >= 2 MDTs [ 1702.670928] Lustre: DEBUG MARKER: == sanity-lfsck test 12b: auto detect Lustre device ====== 16:56:37 (1776200197) [ 1713.633886] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1724.899498] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1734.323978] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1736.681111] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1736.704476] LustreError: Skipped 1 previous similar message [ 1736.718299] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2818 to 0x0:2849 [ 1742.388142] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2671 to 0x0:2721 [ 1745.328658] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1752.291978] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1755.615708] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1771.368561] Lustre: DEBUG MARKER: == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 16:57:45 (1776200265) [ 1772.511461] Lustre: *** cfs_fail_loc=160f, val=0*** [ 1782.243208] Lustre: DEBUG MARKER: == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 16:57:56 (1776200276) [ 1785.823166] Lustre: *** cfs_fail_loc=1610, val=0*** [ 1785.830814] Lustre: Skipped 7 previous similar messages [ 1807.843443] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.853068] Lustre: Skipped 2 previous similar messages [ 1807.858936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1807.864937] Lustre: Skipped 1 previous similar message [ 1813.777905] LustreError: 55969:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200309 with bad export cookie 5102148536754588191 [ 1813.785509] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1813.793703] LustreError: 55969:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1833.161565] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1840.754643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1840.759078] Lustre: Skipped 7 previous similar messages [ 1844.742618] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1855.521396] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1855.522946] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1855.545261] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2946 to 0x0:2977 [ 1863.626227] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2723 to 0x0:2753 [ 1866.048213] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1874.366266] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1890.084807] Lustre: DEBUG MARKER: == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 16:59:44 (1776200384) [ 1895.792162] Lustre: *** cfs_fail_loc=1610, val=0*** [ 1895.794100] Lustre: Skipped 63 previous similar messages [ 1921.169908] Lustre: server umount lustre-MDT0000 complete [ 1921.172056] Lustre: Skipped 7 previous similar messages [ 1925.243824] LustreError: 61468:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200420 with bad export cookie 5102148536754602527 [ 1930.719237] Lustre: 64946:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200420/real 1776200420] req@0000000042275ae5 x1862479433296064/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200426 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1930.760102] Lustre: 64946:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1952.874549] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1969.332965] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1978.464637] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.479499] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3074 to 0x0:3105 [ 1978.486700] LustreError: Skipped 1 previous similar message [ 1980.833174] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1988.695142] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2723 to 0x0:2785 [ 1991.430452] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1998.122468] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2001.611623] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2001.614951] Lustre: Skipped 1 previous similar message [ 2012.317723] Lustre: DEBUG MARKER: == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 17:01:46 (1776200506) [ 2015.855902] Lustre: *** cfs_fail_loc=1611, val=0*** [ 2015.957362] Lustre: *** cfs_fail_loc=1611, val=0*** [ 2015.959660] Lustre: Skipped 63 previous similar messages [ 2026.676839] Lustre: DEBUG MARKER: == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 17:02:01 (1776200521) [ 2028.620462] Lustre: *** cfs_fail_loc=1612, val=0*** [ 2028.623527] Lustre: Skipped 2 previous similar messages [ 2028.711808] Lustre: *** cfs_fail_loc=1612, val=0*** [ 2039.311906] Lustre: DEBUG MARKER: == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 17:02:13 (1776200533) [ 2041.146434] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_15c needs >= 2 MDTs [ 2042.736774] Lustre: DEBUG MARKER: == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 17:02:17 (1776200537) [ 2046.003598] Lustre: *** cfs_fail_loc=1613, val=0*** [ 2054.391758] Lustre: DEBUG MARKER: == sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 17:02:29 (1776200549) [ 2056.524905] Lustre: *** cfs_fail_loc=1614, val=0*** [ 2056.542962] Lustre: Skipped 2 previous similar messages [ 2066.528355] Lustre: DEBUG MARKER: == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 17:02:41 (1776200561) [ 2068.689913] Lustre: *** cfs_fail_loc=1615, val=0*** [ 2068.694341] Lustre: Skipped 1 previous similar message [ 2081.796490] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_18b skipping excluded test 18b [ 2083.761349] Lustre: DEBUG MARKER: == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 17:02:57 (1776200577) [ 2085.795961] Lustre: *** cfs_fail_loc=1617, val=0*** [ 2085.804893] Lustre: Skipped 3 previous similar messages [ 2086.031049] Lustre: *** cfs_fail_loc=1617, val=0*** [ 2086.035438] Lustre: Skipped 1 previous similar message [ 2088.415661] Lustre: *** cfs_fail_loc=1616, val=0*** [ 2088.435378] Lustre: Skipped 3 previous similar messages [ 2107.057405] Lustre: DEBUG MARKER: == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 17:03:20 (1776200600) [ 2110.060017] Lustre: *** cfs_fail_loc=1618, val=0*** [ 2110.061863] Lustre: Skipped 3 previous similar messages [ 2120.672103] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.682734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2120.685613] Lustre: Skipped 3 previous similar messages [ 2120.701264] Lustre: Skipped 1 previous similar message [ 2125.092501] LustreError: 66240:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200620 with bad export cookie 5102148536754617759 [ 2125.097288] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2125.108499] LustreError: 66240:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2125.124862] LustreError: Skipped 1 previous similar message [ 2144.088933] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2156.077416] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2166.225363] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2166.562650] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.575471] LustreError: Skipped 1 previous similar message [ 2166.599115] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3227 to 0x0:3265 [ 2174.632292] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2790 to 0x0:2817 [ 2177.609034] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2185.125849] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2204.803988] Lustre: DEBUG MARKER: == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 17:04:59 (1776200699) [ 2206.743089] Lustre: *** cfs_fail_loc=1618, val=0*** [ 2206.746335] Lustre: Skipped 3 previous similar messages [ 2214.369079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2214.384102] Lustre: Skipped 1 previous similar message [ 2224.782290] LustreError: 73474:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200720 with bad export cookie 5102148536754638619 [ 2246.037953] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2259.531991] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2269.889849] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2270.724620] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3270 to 0x0:3297 [ 2277.824250] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2790 to 0x0:2849 [ 2280.798516] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2288.471503] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2292.549294] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2292.554848] Lustre: Skipped 1 previous similar message [ 2302.541614] LustreError: 80250:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 sleeping for 10000ms [ 2302.554128] LustreError: 80250:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 7 previous similar messages [ 2306.459030] LustreError: 80250:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 2320.203841] Lustre: DEBUG MARKER: == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 17:06:54 (1776200814) [ 2322.207844] Lustre: *** cfs_fail_loc=1616, val=0*** [ 2322.213172] Lustre: Skipped 3 previous similar messages [ 2327.215496] Lustre: *** cfs_fail_loc=161c, val=0*** [ 2342.323316] Lustre: DEBUG MARKER: == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) ========================================================== 17:07:16 (1776200836) [ 2344.469946] Lustre: *** cfs_fail_loc=162e, val=0*** [ 2344.482845] Lustre: Skipped 3 previous similar messages [ 2357.325680] Lustre: DEBUG MARKER: == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 17:07:31 (1776200851) [ 2374.779494] Lustre: DEBUG MARKER: == sanity-lfsck test 19a: OST-object inconsistency self detect ========================================================== 17:07:49 (1776200869) [ 2387.002901] Lustre: DEBUG MARKER: == sanity-lfsck test 19b: OST-object inconsistency self repair ========================================================== 17:08:01 (1776200881) [ 2389.796198] Lustre: *** cfs_fail_loc=1611, val=0*** [ 2389.839586] Lustre: *** cfs_fail_loc=1611, val=0*** [ 2389.842134] Lustre: Skipped 3 previous similar messages [ 2395.691547] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.30@tcp inode [0x200004a51:0x1c:0x0] object 0x0:3307 extent [0-4095], client returned csum 0 (type 4), server csum 703fb3ad (type 4) [ 2396.894142] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.30@tcp inode [0x200004a51:0x1d:0x0] object 0x0:3308 extent [0-4095], client returned csum 0 (type 4), server csum 7cbb935c (type 4) [ 2404.882783] Lustre: DEBUG MARKER: == sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================================================== 17:08:18 (1776200898) [ 2406.854774] Lustre: *** cfs_fail_loc=1616, val=0*** [ 2406.859153] Lustre: Skipped 3 previous similar messages [ 2436.045879] Lustre: DEBUG MARKER: == sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case ========================================================== 17:08:50 (1776200930) [ 2458.368762] Lustre: DEBUG MARKER: == sanity-lfsck test 21: run all LFSCK components by default ========================================================== 17:09:12 (1776200952) [ 2469.939921] Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) ========================================================== 17:09:24 (1776200964) [ 2471.497977] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_22a needs >= 2 MDTs [ 2473.045085] Lustre: DEBUG MARKER: == sanity-lfsck test 22b: LFSCK can repair unmatched pairs (2) ========================================================== 17:09:27 (1776200967) [ 2474.706771] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_22b needs >= 2 MDTs [ 2476.462785] Lustre: DEBUG MARKER: == sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) ========================================================== 17:09:30 (1776200970) [ 2477.763518] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_23a needs >= 2 MDTs [ 2479.384784] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_23b skipping excluded test 23b [ 2480.986417] Lustre: DEBUG MARKER: == sanity-lfsck test 23c: LFSCK can repair dangling name entry (3) ========================================================== 17:09:35 (1776200975) [ 2487.595420] Lustre: *** cfs_fail_loc=1621, val=0*** [ 2487.599373] Lustre: Skipped 2599 previous similar messages [ 2490.165755] LustreError: 86113:0:(osd_index.c:220:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -2 [ 2490.311712] LustreError: 86115:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 sleeping for 10000ms [ 2491.265465] LustreError: 86115:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 2502.221579] Lustre: DEBUG MARKER: == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ========================================================== 17:09:56 (1776200996) [ 2503.978132] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_24 needs >= 2 MDTs [ 2505.321782] Lustre: DEBUG MARKER: == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ========================================================== 17:09:59 (1776200999) [ 2506.865319] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_25 only ldiskfs fixes dirent type [ 2508.719789] Lustre: DEBUG MARKER: == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace ========================================================== 17:10:02 (1776201002) [ 2510.692116] Lustre: *** cfs_fail_loc=1624, val=0*** [ 2510.699720] Lustre: Skipped 1 previous similar message [ 2522.131113] Lustre: DEBUG MARKER: == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========================================================== 17:10:16 (1776201016) [ 2523.783663] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_26b needs >= 2 MDTs [ 2525.880127] Lustre: DEBUG MARKER: == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan ========================================================== 17:10:20 (1776201020) [ 2527.392268] Lustre: *** cfs_fail_loc=1624, val=0*** [ 2538.147558] Lustre: DEBUG MARKER: == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ========================================================== 17:10:32 (1776201032) [ 2540.127243] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_27b needs >= 2 MDTs [ 2542.521191] Lustre: DEBUG MARKER: == sanity-lfsck test 28: Skip the failed MDT(s) when handle orphan MDT-objects ========================================================== 17:10:36 (1776201036) [ 2544.915154] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_28 needs >= 2 MDTs [ 2547.496134] Lustre: DEBUG MARKER: == sanity-lfsck test 29b: LFSCK can repair bad nlink count (2) ========================================================== 17:10:41 (1776201041) [ 2561.572956] Lustre: DEBUG MARKER: == sanity-lfsck test 29c: verify linkEA size limitation == 17:10:55 (1776201055) [ 2583.562653] Lustre: DEBUG MARKER: == sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 17:11:17 (1776201077) [ 2585.688331] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_30 only ldiskfs has lost+found [ 2588.053149] Lustre: DEBUG MARKER: == sanity-lfsck test 31a: The LFSCK can find/repair the name entry with bad name hash (1) ========================================================== 17:11:22 (1776201082) [ 2590.092199] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31a needs >= 2 MDTs [ 2592.149555] Lustre: DEBUG MARKER: == sanity-lfsck test 31b: The LFSCK can find/repair the name entry with bad name hash (2) ========================================================== 17:11:26 (1776201086) [ 2593.928642] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31b needs >= 2 MDTs [ 2595.907390] Lustre: DEBUG MARKER: == sanity-lfsck test 31c: Re-generate the lost master LMV EA for striped directory ========================================================== 17:11:29 (1776201089) [ 2597.652671] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31c needs >= 2 MDTs [ 2599.638807] Lustre: DEBUG MARKER: == sanity-lfsck test 31d: Set broken striped directory (modified after broken) as read-only ========================================================== 17:11:33 (1776201093) [ 2601.061701] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31d needs >= 2 MDTs [ 2602.621414] Lustre: DEBUG MARKER: == sanity-lfsck test 31e: Re-generate the lost slave LMV EA for striped directory (1) ========================================================== 17:11:37 (1776201097) [ 2604.110893] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31e needs >= 2 MDTs [ 2606.011617] Lustre: DEBUG MARKER: == sanity-lfsck test 31f: Re-generate the lost slave LMV EA for striped directory (2) ========================================================== 17:11:40 (1776201100) [ 2607.334408] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31f needs >= 2 MDTs [ 2609.164386] Lustre: DEBUG MARKER: == sanity-lfsck test 31g: Repair the corrupted slave LMV EA ========================================================== 17:11:43 (1776201103) [ 2610.513597] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31g needs >= 2 MDTs [ 2612.422930] Lustre: DEBUG MARKER: == sanity-lfsck test 31h: Repair the corrupted shard's name entry ========================================================== 17:11:46 (1776201106) [ 2613.933712] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31h needs >= 2 MDTs [ 2615.691127] Lustre: DEBUG MARKER: == sanity-lfsck test 32a: stop LFSCK when some OST failed ========================================================== 17:11:50 (1776201110) [ 2626.857717] LustreError: 89887:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 162d sleeping for 3000ms [ 2629.641054] Lustre: Failing over lustre-OST0000 [ 2629.884416] Lustre: server umount lustre-OST0000 complete [ 2629.889115] Lustre: Skipped 8 previous similar messages [ 2629.944527] LustreError: 89887:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 162d awake [ 2629.953630] LustreError: 89887:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2629.962222] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 2629.970420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.979043] Lustre: Skipped 2 previous similar messages [ 2629.983200] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.993585] LustreError: Skipped 3 previous similar messages [ 2643.009696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2643.011848] Lustre: Skipped 11 previous similar messages [ 2643.032584] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2644.195927] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2644.410005] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2644.410709] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [ 2644.446045] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3418 to 0x0:3457 [ 2644.449867] Lustre: Skipped 1 previous similar message [ 2648.744736] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2657.972622] Lustre: DEBUG MARKER: == sanity-lfsck test 32b: stop LFSCK when some MDT failed ========================================================== 17:12:32 (1776201152) [ 2660.105932] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_32b needs >= 2 MDTs [ 2662.420884] Lustre: DEBUG MARKER: == sanity-lfsck test 33: check LFSCK paramters =========== 17:12:36 (1776201156) [ 2675.533325] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2696.437482] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2721.274535] Lustre: DEBUG MARKER: == sanity-lfsck test 34: LFSCK can rebuild the lost agent object ========================================================== 17:13:35 (1776201215) [ 2722.453841] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_34 needs >= 2 MDTs [ 2723.927221] Lustre: DEBUG MARKER: == sanity-lfsck test 35: LFSCK can rebuild the lost agent entry ========================================================== 17:13:38 (1776201218) [ 2725.159370] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_35 needs >= 2 MDTs [ 2726.831816] Lustre: DEBUG MARKER: == sanity-lfsck test 36a: rebuild LOV EA for mirrored file (1) ========================================================== 17:13:41 (1776201221) [ 2728.052640] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36a needs >= 3 OSTs [ 2729.713315] Lustre: DEBUG MARKER: == sanity-lfsck test 36b: rebuild LOV EA for mirrored file (2) ========================================================== 17:13:44 (1776201224) [ 2731.068529] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36b needs >= 3 OSTs [ 2732.558832] Lustre: DEBUG MARKER: == sanity-lfsck test 36c: rebuild LOV EA for mirrored file (3) ========================================================== 17:13:47 (1776201227) [ 2733.917653] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36c needs >= 3 OSTs [ 2735.252875] Lustre: DEBUG MARKER: == sanity-lfsck test 37: LFSCK must skip a ORPHAN ======== 17:13:49 (1776201229) [ 2744.412620] Lustre: DEBUG MARKER: == sanity-lfsck test 38: LFSCK does not break foreign file and reverse is also true ========================================================== 17:13:58 (1776201238) [ 2757.254700] Lustre: DEBUG MARKER: == sanity-lfsck test 39: LFSCK does not break foreign dir and reverse is also true ========================================================== 17:14:11 (1776201251) [ 2767.434352] Lustre: DEBUG MARKER: == sanity-lfsck test 40a: LFSCK correctly fixes lmm_oi in composite layout ========================================================== 17:14:22 (1776201262) [ 2768.529123] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_40a needs >= 2 MDTs [ 2769.881411] Lustre: DEBUG MARKER: == sanity-lfsck test 41: SEL support in LFSCK ============ 17:14:24 (1776201264) [ 2799.890772] Lustre: DEBUG MARKER: == sanity-lfsck test 42: LFSCK can repair inconsistent MDT-object/OST-object encryption flags ========================================================== 17:14:53 (1776201293) [ 2802.001935] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_42 skip ZFS backend [ 2814.290951] LustreError: 77860:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201309 with bad export cookie 5102148536754641153 [ 2814.303331] LustreError: 77860:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2814.306878] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2814.320464] LustreError: Skipped 1 previous similar message [ 2818.975276] Lustre: 12167:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201307/real 1776201307] req@0000000028041f5a x1862479433521792/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201314 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 2837.934978] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2841.324968] Key type lgssc unregistered [ 2841.546824] LNet: 96934:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2841.563305] LNet: Removed LNI 192.168.202.130@tcp [ 2842.205929] Key type .llcrypt unregistered [ 2842.209248] Key type ._llcrypt unregistered [ 2857.407235] alg: No test for adler32 (adler32-zlib) [ 2858.157992] Key type ._llcrypt registered [ 2858.160137] Key type .llcrypt registered [ 2858.249579] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 2869.332785] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2870.333134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2870.565162] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 2870.755504] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2870.758836] LNet: Accept secure, port 988 [ 2872.415120] Key type lgssc registered [ 2873.240082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2881.007369] vdc: vdc1 vdc9 [ 2889.786597] vde: vde1 vde9 [ 2900.246897] vdf: vdf1 vdf9 [ 2917.624177] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2926.362453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2926.589404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2926.656225] Lustre: lustre-MDT0000: new disk, initializing [ 2926.959616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.028273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2930.936258] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2940.001257] Lustre: lustre-OST0000: new disk, initializing [ 2940.015461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2940.018613] Lustre: Skipped 1 previous similar message [ 2940.136424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2944.511391] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2953.514674] Lustre: lustre-OST0001: new disk, initializing [ 2953.517725] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2953.598490] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2958.919208] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2970.620758] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2978.811109] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2985.546464] Lustre: DEBUG MARKER: == sanity-lfsck test complete, duration 2765 sec ========= 17:17:59 (1776201479) [ 2990.572571] Lustre: server umount lustre-MDT0000 complete [ 2994.792839] LustreError: 100794:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201490 with bad export cookie 17651909211913571955 [ 2994.802281] LustreError: 166-1: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3001.311138] Lustre: 103540:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201490/real 1776201490] req@0000000041bc6db7 x1862482123572608/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201496 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3001.375427] Lustre: server umount lustre-OST0000 complete [ 3001.807221] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.994794] Lustre: server umount lustre-OST0001 complete [ 3016.163376] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3019.215247] Key type lgssc unregistered [ 3019.445478] LNet: 104346:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3019.455265] LNet: Removed LNI 192.168.202.130@tcp [ 3019.998681] Key type .llcrypt unregistered [ 3020.002337] Key type ._llcrypt unregistered