[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479215427 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001022] APIC: Switch to symmetric I/O mode setup [ 0.003183] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005020] kvm-guest: setup PV IPIs [ 0.007884] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010126] LSM: Security Framework initializing [ 0.011028] Yama: becoming mindful. [ 0.012024] SELinux: Initializing. [ 0.013055] *** VALIDATE selinux *** [ 0.020210] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024708] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026215] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028137] *** VALIDATE tmpfs *** [ 0.030404] *** VALIDATE proc *** [ 0.031250] *** VALIDATE cgroup *** [ 0.032005] *** VALIDATE cgroup2 *** [ 0.033270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035029] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037040] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.041000] debug: unmapping init [mem 0xffffffffb0e59000-0xffffffffb0e60fff] [ 0.043178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044647] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045021] ... version: 2 [ 0.045781] ... bit width: 48 [ 0.046016] ... generic registers: 4 [ 0.047016] ... value mask: 0000ffffffffffff [ 0.048017] ... max period: 00007fffffffffff [ 0.049021] ... fixed-purpose events: 3 [ 0.050015] ... event mask: 000000070000000f [ 0.051386] rcu: Hierarchical SRCU implementation. [ 0.053581] smp: Bringing up secondary CPUs ... [ 0.054538] x86: Booting SMP configuration: [ 0.055017] .... node #0, CPUs: #1 #2 #3 [ 0.058155] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061033] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.257041] node 0 deferred pages initialised in 194ms [ 0.260466] devtmpfs: initialized [ 0.261228] x86/mm: Memory block size: 128MB [ 0.263788] gcov: version magic: 0x41383552 [ 0.265573] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.269084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.271319] pinctrl core: initialized pinctrl subsystem [ 0.273128] [ 0.273716] ************************************************************* [ 0.276010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277011] ** ** [ 0.279008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.281008] ** ** [ 0.282007] ** This means that this kernel is built to expose internal ** [ 0.284007] ** IOMMU data structures, which may compromise security on ** [ 0.286008] ** your system. ** [ 0.288007] ** ** [ 0.289008] ** If you see this message and you are not debugging the ** [ 0.291009] ** kernel, report this immediately to your vendor! ** [ 0.294012] ** ** [ 0.296011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.297008] ************************************************************* [ 0.299992] NET: Registered protocol family 16 [ 0.302490] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.305051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.307050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.310146] cpuidle: using governor menu [ 0.311714] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.313444] PCI: Using configuration type 1 for base access [ 0.315125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.327311] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.329065] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.331306] cryptd: max_cpu_qlen set to 1000 [ 0.334369] ACPI: Added _OSI(Module Device) [ 0.335023] ACPI: Added _OSI(Processor Device) [ 0.336008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.338011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.341268] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.347582] ACPI: Interpreter enabled [ 0.348000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.350014] ACPI: Using IOAPIC for interrupt routing [ 0.351091] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.354362] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.366000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.366000] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.369018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.372092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.376162] acpiphp: Slot [2] registered [ 0.377098] acpiphp: Slot [3] registered [ 0.379075] acpiphp: Slot [4] registered [ 0.380079] acpiphp: Slot [5] registered [ 0.381102] acpiphp: Slot [6] registered [ 0.382074] acpiphp: Slot [7] registered [ 0.383099] acpiphp: Slot [8] registered [ 0.385126] acpiphp: Slot [9] registered [ 0.386094] acpiphp: Slot [10] registered [ 0.387090] acpiphp: Slot [11] registered [ 0.389081] acpiphp: Slot [12] registered [ 0.389984] acpiphp: Slot [13] registered [ 0.391075] acpiphp: Slot [14] registered [ 0.392113] acpiphp: Slot [15] registered [ 0.393066] acpiphp: Slot [16] registered [ 0.395073] acpiphp: Slot [17] registered [ 0.396078] acpiphp: Slot [18] registered [ 0.397000] acpiphp: Slot [19] registered [ 0.397000] acpiphp: Slot [20] registered [ 0.399083] acpiphp: Slot [21] registered [ 0.400112] acpiphp: Slot [22] registered [ 0.400988] acpiphp: Slot [23] registered [ 0.402052] acpiphp: Slot [24] registered [ 0.402930] acpiphp: Slot [25] registered [ 0.403074] acpiphp: Slot [26] registered [ 0.404063] acpiphp: Slot [27] registered [ 0.404901] acpiphp: Slot [28] registered [ 0.405000] acpiphp: Slot [29] registered [ 0.405000] acpiphp: Slot [30] registered [ 0.406081] acpiphp: Slot [31] registered [ 0.407081] PCI host bridge to bus 0000:00 [ 0.408030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.410035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.412029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.414031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.417029] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.419025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.420184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.424211] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.428645] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.437529] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.440382] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.442018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.443016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.445012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.447502] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.449807] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.452040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.455655] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.458011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.468020] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.472016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.475458] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.484044] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.493024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.510016] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.519547] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.530022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.541025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.566047] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.576914] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.586017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.592024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.616025] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.627000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.644027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.653028] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.673035] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.685000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.692018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.699023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.718021] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.731940] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.766015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.769000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.783021] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.792682] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.795089] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.797361] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.799262] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.800163] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.805252] iommu: Default domain type: Passthrough [ 0.807848] SCSI subsystem initialized [ 0.809107] ACPI: bus type USB registered [ 0.811170] usbcore: registered new interface driver usbfs [ 0.813111] usbcore: registered new interface driver hub [ 0.814057] usbcore: registered new device driver usb [ 0.816177] pps_core: LinuxPPS API ver. 1 registered [ 0.818012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.822224] PTP clock support registered [ 0.826057] EDAC MC: Ver: 3.0.0 [ 0.827481] PCI: Using ACPI for IRQ routing [ 0.829847] NetLabel: Initializing [ 0.830008] NetLabel: domain hash size = 128 [ 0.832007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.833084] NetLabel: unlabeled traffic allowed by default [ 0.836063] vgaarb: loaded [ 0.837269] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.839010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.852276] clocksource: Switched to clocksource kvm-clock [ 0.966487] VFS: Disk quotas dquot_6.6.0 [ 0.967935] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.970364] *** VALIDATE ramfs *** [ 0.971486] *** VALIDATE hugetlbfs *** [ 0.973510] pnp: PnP ACPI init [ 0.975707] pnp: PnP ACPI: found 6 devices [ 0.993715] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.996792] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.998847] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.001133] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.003587] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.006223] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.008910] NET: Registered protocol family 2 [ 1.011377] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.016455] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.019337] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.024357] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.027363] TCP: Hash tables configured (established 65536 bind 65536) [ 1.029750] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.032187] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.034409] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.036639] NET: Registered protocol family 1 [ 1.038855] RPC: Registered named UNIX socket transport module. [ 1.040890] RPC: Registered udp transport module. [ 1.042484] RPC: Registered tcp transport module. [ 1.043829] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.045761] NET: Registered protocol family 44 [ 1.047224] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.049142] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.050917] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.052751] PCI: CLS 0 bytes, default 64 [ 1.054216] Unpacking initramfs... [ 2.608388] debug: unmapping init [mem 0xffff88b7bcc54000-0xffff88b7bffbffff] [ 2.613184] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.615406] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.618212] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.274220] Initialise system trusted keyrings [ 3.275884] Key type blacklist registered [ 3.277895] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.286773] zbud: loaded [ 3.291028] *** VALIDATE nfs *** [ 3.292323] *** VALIDATE nfs4 *** [ 3.293953] pstore: using deflate compression [ 3.297820] Platform Keyring initialized [ 3.478659] NET: Registered protocol family 38 [ 3.480151] Key type asymmetric registered [ 3.484224] Asymmetric key parser 'x509' registered [ 3.487904] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.499407] io scheduler mq-deadline registered [ 3.503334] io scheduler kyber registered [ 3.506708] io scheduler bfq registered [ 3.512632] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.517212] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.519767] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.522447] ACPI: Power Button [PWRF] [ 3.671256] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.879558] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.122137] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.263138] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.532230] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.576869] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.619703] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.624576] Non-volatile memory driver v1.3 [ 4.626092] Linux agpgart interface v0.103 [ 4.667692] virtio_blk virtio1: [vda] 132696 512-byte logical blocks (67.9 MB/64.8 MiB) [ 4.670508] vda: detected capacity change from 0 to 67940352 [ 4.688116] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.692732] vdb: detected capacity change from 0 to 1073741824 [ 4.714348] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.716965] vdc: detected capacity change from 0 to 2621440000 [ 4.733377] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.740521] vdd: detected capacity change from 0 to 2621440000 [ 4.758689] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.765057] vde: detected capacity change from 0 to 4294967296 [ 4.784363] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.787375] vdf: detected capacity change from 0 to 4294967296 [ 4.795380] libphy: Fixed MDIO Bus: probed [ 4.804527] usbcore: registered new interface driver usbserial_generic [ 4.808310] usbserial: USB Serial support registered for generic [ 4.810418] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.816558] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.818095] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.823858] mousedev: PS/2 mouse device common for all mice [ 4.832114] rtc_cmos 00:05: RTC can wake from S4 [ 4.838887] rtc_cmos 00:05: registered as rtc0 [ 4.840319] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.849992] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.851506] intel_pstate: CPU model not supported [ 4.861352] hid: raw HID events driver (C) Jiri Kosina [ 4.874513] usbcore: registered new interface driver usbhid [ 4.877373] usbhid: USB HID core driver [ 4.891188] drop_monitor: Initializing network drop monitor service [ 4.897480] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.902892] Initializing XFRM netlink socket [ 4.903260] NET: Registered protocol family 10 [ 4.908711] Segment Routing with IPv6 [ 4.908757] NET: Registered protocol family 17 [ 4.910865] mpls_gso: MPLS GSO support [ 4.939676] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.966370] RAS: Correctable Errors collector initialized. [ 4.971792] AVX version of gcm_enc/dec engaged. [ 4.972880] AES CTR mode by8 optimization enabled [ 5.249572] sched_clock: Marking stable (5249555878, 0)->(6144549862, -894993984) [ 5.256391] registered taskstats version 1 [ 5.261493] Loading compiled-in X.509 certificates [ 5.272336] zswap: loaded using pool lzo/zbud [ 5.343500] Key type big_key registered [ 5.395062] Key type encrypted registered [ 5.396396] ima: No TPM chip found, activating TPM-bypass! [ 5.405385] ima: Allocated hash algorithm: sha1 [ 5.412938] ima: No architecture policies found [ 5.414264] evm: Initialising EVM extended attributes: [ 5.422140] evm: security.selinux [ 5.424436] evm: security.ima [ 5.430882] evm: security.capability [ 5.432090] evm: HMAC attrs: 0x1 [ 5.440273] rtc_cmos 00:05: setting system clock to 2025-07-16 17:41:50 UTC (1752687710) [ 5.456408] debug: unmapping init [mem 0xffffffffb1e03000-0xffffffffb1ffffff] [ 5.464176] debug: unmapping init [mem 0xffffffffb0b82000-0xffffffffb0e58fff] [ 5.483228] Write protecting the kernel read-only data: 28672k [ 5.491288] debug: unmapping init [mem 0xffffffffaf203000-0xffffffffaf3fffff] [ 5.499823] debug: unmapping init [mem 0xffffffffafb14000-0xffffffffafbfffff] [ 5.600945] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.624646] systemd[1]: Detected virtualization kvm. [ 5.631162] systemd[1]: Detected architecture x86-64. [ 5.636347] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.698072] systemd[1]: No hostname configured. [ 5.699497] systemd[1]: Set hostname to . [ 5.701166] random: systemd: uninitialized urandom read (16 bytes read) [ 5.711914] systemd[1]: Initializing machine ID from random generator. [ 5.882352] random: ln: uninitialized urandom read (6 bytes read) [ 6.188783] random: systemd: uninitialized urandom read (16 bytes read) [ 6.191122] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.207643] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.218728] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.526575] device-mapper: uevent: version 1.0.3 [ 7.528465] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.690129] virtio_net virtio0 ens2: renamed from eth0 [ 8.805687] random: fast init done [ 8.861150] scsi host0: ata_piix [ 8.894707] scsi host1: ata_piix [ 8.899368] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.906143] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.994175] random: crng init done [ 13.999226] random: 7 urandom warning(s) missed due to ratelimiting [ 16.150723] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.719251] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.587677] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.088663] SELinux: Disabled at runtime. [ 20.170795] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.179316] systemd[1]: Detected virtualization kvm. [ 20.180575] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.170397] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.173910] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.181687] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.187222] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.190553] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.201509] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.218456] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ 21.442199] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 22.870512] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.853363] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.888763] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.331432] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.533145] EDAC sbridge: Ver: 1.1.2 [ 28.503261] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 29.144415] NFS: Registering the id_resolver key type [ 29.149529] Key type id_resolver registered [ 29.153067] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg357-server login: [ 49.450382] spl: loading out-of-tree module taints kernel. [ 51.626690] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 55.303417] Key type ._llcrypt registered [ 55.305018] Key type .llcrypt registered [ 55.343213] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 61.935074] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 62.421057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.445800] alg: No test for adler32 (adler32-zlib) [ 63.430871] Lustre: Lustre: Build Version: 2.16.57_1_gf739d57 [ 63.742097] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 63.743863] LNet: Accept secure, port 988 [ 65.343170] Key type lgssc registered [ 65.773959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.944295] vdc: vdc1 vdc9 [ 72.013792] vde: vde1 vde9 [ 75.146672] vdf: vdf1 vdf9 [ 80.797765] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 83.789584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 84.898564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 84.988348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 85.028988] Lustre: lustre-MDT0000: new disk, initializing [ 85.126780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 85.147227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 86.484199] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.831140] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 90.721222] Lustre: lustre-OST0000: new disk, initializing [ 90.723724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 90.725983] Lustre: Skipped 1 previous similar message [ 90.750983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 92.748874] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 94.674359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 94.676538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 94.710585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 96.738849] Lustre: lustre-OST0001: new disk, initializing [ 96.740376] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 96.765640] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 98.866521] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 104.565982] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 105.000929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 105.003824] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 105.037458] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 110.199926] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 117.114404] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing check_logdir /tmp/testlogs/ [ 118.651438] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing yml_node [ 120.125709] Lustre: DEBUG MARKER: Client: 2.16.57.1 [ 120.928130] Lustre: DEBUG MARKER: MDS: 2.16.57.1 [ 121.689696] Lustre: DEBUG MARKER: OSS: 2.16.57.1 [ 122.175259] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-benchmark ============----- Wed Jul 16 13:43:47 EDT 2025 [ 127.608099] Lustre: DEBUG MARKER: excepting tests: [ 128.101605] Lustre: DEBUG MARKER: skipping tests SLOW=no: iozone [ 128.626429] Lustre: DEBUG MARKER: === sanity-benchmark: start setup 13:43:54 (1752687834) === [ 129.830123] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing check_config_client /mnt/lustre [ 135.272454] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 136.379344] Lustre: 10954:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 137.466883] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 138.614764] Lustre: DEBUG MARKER: === sanity-benchmark: finish setup 13:44:04 (1752687844) === [ 139.125717] Lustre: DEBUG MARKER: == sanity-benchmark test dbench: dbench ================== 13:44:04 (1752687844) [ 141.347161] Lustre: DEBUG MARKER: == sanity-benchmark test bonnie: bonnie++ ================ 13:44:07 (1752687847) [ 141.877040] Lustre: DEBUG MARKER: min OST has 3766272kB available, using 5649408kB file size [ 166.366079] ------------[ cut here ]------------ [ 166.366079] ------------[ cut here ]------------ [ 166.366102] list_del corruption. next->prev should be ffff88b702d7a858, but was ffffffffc18894f0 [ 166.368342] list_del corruption. next->prev should be ffff88b702d7a858, but was ffffffffc18894f0 [ 166.370040] WARNING: CPU: 2 PID: 8002 at lib/list_debug.c:56 __list_del_entry_valid+0x122/0x140 [ 166.374063] WARNING: CPU: 1 PID: 6323 at lib/list_debug.c:56 __list_del_entry_valid+0x122/0x140 [ 166.378443] Modules linked in: [ 166.382104] Modules linked in: lustre(O) [ 166.385752] lustre(O) [ 166.387029] osp(O) [ 166.388243] osp(O) [ 166.389151] ofd(O) [ 166.389835] ofd(O) [ 166.390521] lod(O) [ 166.390892] lod(O) [ 166.391402] mdt(O) [ 166.391829] mdt(O) [ 166.392326] mdd(O) [ 166.393007] mdd(O) [ 166.393531] mgs(O) [ 166.393945] mgs(O) [ 166.394419] osd_zfs(O) [ 166.395272] osd_zfs(O) [ 166.395835] lquota(O) [ 166.396626] lquota(O) [ 166.397356] lfsck(O) [ 166.397868] lfsck(O) [ 166.398907] obdecho(O) [ 166.399523] obdecho(O) [ 166.400244] mgc(O) [ 166.400748] mgc(O) [ 166.401236] mdc(O) [ 166.402045] mdc(O) [ 166.402584] lov(O) [ 166.403055] lov(O) [ 166.403498] osc(O) [ 166.403899] osc(O) [ 166.404571] lmv(O) [ 166.405899] lmv(O) [ 166.406945] fid(O) [ 166.407555] fid(O) [ 166.409482] fld(O) [ 166.410635] fld(O) [ 166.412433] ptlrpc_gss(O) [ 166.413378] ptlrpc_gss(O) [ 166.414427] ptlrpc(O) [ 166.416393] ptlrpc(O) [ 166.417207] obdclass(O) [ 166.419565] obdclass(O) [ 166.420758] ksocklnd(O) [ 166.423048] ksocklnd(O) [ 166.423998] lnet(O) [ 166.425929] lnet(O) [ 166.427140] libcfs(O) [ 166.428338] libcfs(O) [ 166.429194] zfs(O) [ 166.430140] zfs(O) [ 166.432138] spl(O) [ 166.433390] spl(O) [ 166.434215] rpcsec_gss_krb5 [ 166.435455] rpcsec_gss_krb5 [ 166.436243] auth_rpcgss [ 166.436963] auth_rpcgss [ 166.438030] nfsv4 [ 166.438775] nfsv4 [ 166.439551] dns_resolver [ 166.440893] dns_resolver [ 166.441774] intel_rapl_msr [ 166.442576] intel_rapl_msr [ 166.443139] intel_rapl_common [ 166.443775] intel_rapl_common [ 166.444266] sb_edac [ 166.445210] sb_edac [ 166.445795] rapl [ 166.446241] rapl [ 166.446679] pcspkr [ 166.447207] pcspkr [ 166.448584] i2c_piix4 [ 166.449892] i2c_piix4 [ 166.450290] squashfs [ 166.451185] squashfs [ 166.451944] crct10dif_pclmul [ 166.452384] crct10dif_pclmul [ 166.452815] crc32_pclmul [ 166.453868] crc32_pclmul [ 166.454760] crc32c_intel [ 166.456711] crc32c_intel [ 166.457784] ata_generic [ 166.459294] ata_generic [ 166.459769] ghash_clmulni_intel [ 166.460635] ghash_clmulni_intel [ 166.461087] serio_raw ata_piix [ 166.462453] serio_raw [ 166.463012] libata [ 166.464269] ata_piix [ 166.466402] dm_mirror [ 166.466980] libata [ 166.467692] dm_region_hash [ 166.468493] dm_mirror [ 166.468960] dm_log [ 166.469567] dm_region_hash [ 166.470061] dm_mod [ 166.470437] dm_log dm_mod [ 166.471081] sha512_ssse3 [ 166.471471] sha512_ssse3 [ 166.472060] sha512_generic [ 166.472523] sha512_generic [ 166.473033] [ 166.474130] [ 166.475992] CPU: 1 PID: 6323 Comm: z_wr_iss Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 166.476468] CPU: 2 PID: 8002 Comm: z_wr_iss Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 166.477107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 166.479976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 166.482354] RIP: 0010:__list_del_entry_valid+0x122/0x140 [ 166.485015] RIP: 0010:__list_del_entry_valid+0x122/0x140 [ 166.487349] Code: 0c 03 01 31 c0 48 83 05 eb 7c 0c 03 01 c3 48 c7 c7 30 e2 9e af 48 83 05 eb 7c 0c 03 01 e8 c2 48 8f ff 48 83 05 e6 7c 0c 03 01 <0f> 0b 48 83 05 e4 7c 0c 03 01 48 83 05 e4 7c 0c 03 01 31 c0 48 83 [ 166.488569] Code: 0c 03 01 31 c0 48 83 05 eb 7c 0c 03 01 c3 48 c7 c7 30 e2 9e af 48 83 05 eb 7c 0c 03 01 e8 c2 48 8f ff 48 83 05 e6 7c 0c 03 01 <0f> 0b 48 83 05 e4 7c 0c 03 01 48 83 05 e4 7c 0c 03 01 31 c0 48 83 [ 166.489949] RSP: 0018:ffffa1be4334b7f0 EFLAGS: 00010202 [ 166.495374] RSP: 0018:ffffa1be430977f0 EFLAGS: 00010202 [ 166.501109] [ 166.503855] [ 166.503859] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 166.505486] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 166.505822] RDX: ffff88b84212e640 RSI: ffff88b84211e5a8 RDI: ffff88b84211e5a8 [ 166.506185] RDX: ffff88b8420ae640 RSI: ffff88b84209e5a8 RDI: ffff88b84209e5a8 [ 166.508367] RBP: ffff88b702d7a858 R08: 2d2d2d2d2d2d2d2d R09: 656820747563205b [ 166.510112] RBP: ffff88b702d7a858 R08: 2e6e6f6974707572 R09: 726f63206c65645f [ 166.513687] R10: 000000002d2d2d2d R11: 000000002d2d2d2d R12: ffffffffc18894f0 [ 166.515775] R10: 74707572726f6320 R11: 6c65645f7473696c R12: ffffffffc18894f0 [ 166.518450] R13: ffff88b702d7a800 R14: 0000000000000000 R15: 0000000000600210 [ 166.520172] R13: ffff88b702d7a800 R14: 0000000000000000 R15: 0000000000600210 [ 166.523540] FS: 0000000000000000(0000) GS:ffff88b842100000(0000) knlGS:0000000000000000 [ 166.525362] FS: 0000000000000000(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 166.527534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 166.528979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 166.530963] CR2: 00007fc41bc0f3e0 CR3: 0000000038416005 CR4: 0000000000170ee0 [ 166.534496] CR2: 00007f372864b000 CR3: 0000000038416002 CR4: 0000000000170ee0 [ 166.536394] Call Trace: [ 166.537708] Call Trace: [ 166.539141] ? show_regs.cold.9+0x22/0x2f [ 166.541144] ? show_regs.cold.9+0x22/0x2f [ 166.541857] ? __warn+0xc8/0x150 [ 166.543406] ? __warn+0xc8/0x150 [ 166.544521] ? __list_del_entry_valid+0x122/0x140 [ 166.545663] ? __list_del_entry_valid+0x122/0x140 [ 166.546280] ? report_bug+0x113/0x140 [ 166.547034] ? report_bug+0x113/0x140 [ 166.547915] ? do_error_trap+0xb6/0x130 [ 166.548820] ? do_error_trap+0xb6/0x130 [ 166.549583] ? do_invalid_op+0x46/0x60 [ 166.550907] ? do_invalid_op+0x46/0x60 [ 166.553136] ? __list_del_entry_valid+0x122/0x140 [ 166.554846] ? __list_del_entry_valid+0x122/0x140 [ 166.556110] ? invalid_op+0x14/0x20 [ 166.557345] ? invalid_op+0x14/0x20 [ 166.558720] ? __list_del_entry_valid+0x122/0x140 [ 166.559691] ? __list_del_entry_valid+0x122/0x140 [ 166.560898] ldlm_namespace_move_to_active_locked+0x36/0xd0 [ptlrpc] [ 166.562878] ldlm_namespace_move_to_active_locked+0x36/0xd0 [ptlrpc] [ 166.564855] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 166.565943] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 166.567773] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 166.569276] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 166.570397] do_shrink_slab+0x41/0x530 [ 166.571558] do_shrink_slab+0x41/0x530 [ 166.574342] shrink_slab+0xff/0x4f0 [ 166.576018] shrink_slab+0xff/0x4f0 [ 166.577264] shrink_node+0x321/0xc20 [ 166.578703] shrink_node+0x321/0xc20 [ 166.579816] do_try_to_free_pages+0xfc/0x7a0 [ 166.581262] do_try_to_free_pages+0xfc/0x7a0 [ 166.582631] try_to_free_pages+0x144/0x330 [ 166.584425] try_to_free_pages+0x144/0x330 [ 166.586067] __alloc_pages_slowpath+0x59b/0x1870 [ 166.586998] __alloc_pages_slowpath+0x59b/0x1870 [ 166.588390] ? _cond_resched+0x1d/0x50 [ 166.590072] ? _cond_resched+0x1d/0x50 [ 166.591359] ? do_raw_spin_unlock+0x75/0x190 [ 166.594587] ? do_raw_spin_unlock+0x75/0x190 [ 166.596863] ? get_page_from_freelist+0x114/0x5d0 [ 166.597758] ? get_page_from_freelist+0x114/0x5d0 [ 166.598967] __alloc_pages_nodemask+0x4d3/0x550 [ 166.600549] __alloc_pages_nodemask+0x4d3/0x550 [ 166.602254] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 166.605272] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 166.607006] ? __mutex_init+0x67/0x80 [ 166.608587] ? __mutex_init+0x67/0x80 [ 166.610016] abd_alloc+0x5d/0x100 [zfs] [ 166.611233] abd_alloc+0x5d/0x100 [zfs] [ 166.612115] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 166.614704] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 166.616166] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 166.617738] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 166.619044] arc_write_ready+0x3c7/0x7b0 [zfs] [ 166.620202] arc_write_ready+0x3c7/0x7b0 [zfs] [ 166.621756] zio_ready+0x5b/0x510 [zfs] [ 166.624742] zio_ready+0x5b/0x510 [zfs] [ 166.626709] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 166.628697] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 166.629945] zio_execute+0xbb/0x170 [zfs] [ 166.631820] zio_execute+0xbb/0x170 [zfs] [ 166.635210] taskq_thread+0x45c/0xbc0 [spl] [ 166.636955] taskq_thread+0x45c/0xbc0 [spl] [ 166.637734] ? try_to_wake_up+0x710/0x710 [ 166.638552] ? try_to_wake_up+0x710/0x710 [ 166.639326] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 166.640207] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 166.640952] ? taskq_thread_spawn+0x90/0x90 [spl] [ 166.642511] ? taskq_thread_spawn+0x90/0x90 [spl] [ 166.645908] kthread+0x1d1/0x200 [ 166.647294] kthread+0x1d1/0x200 [ 166.648380] ? set_kthread_struct+0x70/0x70 [ 166.649603] ? set_kthread_struct+0x70/0x70 [ 166.650447] ret_from_fork+0x1f/0x30 [ 166.651612] ret_from_fork+0x1f/0x30 [ 166.653808] ---[ end trace 2e86c68e9c14ac27 ]--- [ 166.654853] ---[ end trace 2e86c68e9c14ac28 ]--- [ 166.660164] ------------[ cut here ]------------ [ 166.662462] list_add double add: new=ffff88b702d7a858, prev=ffff88b702d7a858, next=ffffffffc18894f0. [ 166.666004] WARNING: CPU: 1 PID: 6323 at lib/list_debug.c:31 __list_add_valid+0xd7/0x100 [ 166.669706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 166.698583] CPU: 1 PID: 6323 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 166.702505] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 166.705422] RIP: 0010:__list_add_valid+0xd7/0x100 [ 166.708016] Code: 7b 0c 03 01 c3 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 10 e3 9e af 48 83 05 4e 7b 0c 03 01 e8 cd 47 8f ff 48 83 05 49 7b 0c 03 01 <0f> 0b 48 83 05 47 7b 0c 03 01 48 83 05 47 7b 0c 03 01 31 c0 48 83 [ 166.712805] RSP: 0018:ffffa1be4334b7f0 EFLAGS: 00010202 [ 166.714210] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 166.718283] RDX: ffff88b8420ae640 RSI: ffff88b84209e5a8 RDI: ffff88b84209e5a8 [ 166.721257] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 166.723662] R10: 0000000000000001 R11: ffffa1be4334b5e0 R12: ffffffffc18894f0 [ 166.725995] R13: ffff88b702d7a858 R14: 0000000000000000 R15: 0000000000600210 [ 166.729361] FS: 0000000000000000(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 166.731179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 166.732478] CR2: 00007f372864b000 CR3: 0000000038416002 CR4: 0000000000170ee0 [ 166.734118] Call Trace: [ 166.735046] ? show_regs.cold.9+0x22/0x2f [ 166.738265] ? __warn+0xc8/0x150 [ 166.739646] ? __list_add_valid+0xd7/0x100 [ 166.741173] ? report_bug+0x113/0x140 [ 166.742478] ? do_error_trap+0xb6/0x130 [ 166.744059] ? do_invalid_op+0x46/0x60 [ 166.746050] ? __list_add_valid+0xd7/0x100 [ 166.747826] ? invalid_op+0x14/0x20 [ 166.748627] ? __list_add_valid+0xd7/0x100 [ 166.750271] ldlm_namespace_move_to_active_locked+0x64/0xd0 [ptlrpc] [ 166.752102] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 166.753556] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 166.756420] do_shrink_slab+0x41/0x530 [ 166.758034] shrink_slab+0xff/0x4f0 [ 166.759417] shrink_node+0x321/0xc20 [ 166.760746] do_try_to_free_pages+0xfc/0x7a0 [ 166.762216] try_to_free_pages+0x144/0x330 [ 166.763122] __alloc_pages_slowpath+0x59b/0x1870 [ 166.764306] ? _cond_resched+0x1d/0x50 [ 166.765340] ? do_raw_spin_unlock+0x75/0x190 [ 166.768017] ? get_page_from_freelist+0x114/0x5d0 [ 166.769255] __alloc_pages_nodemask+0x4d3/0x550 [ 166.770328] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 166.772000] ? __mutex_init+0x67/0x80 [ 166.772966] abd_alloc+0x5d/0x100 [zfs] [ 166.774632] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 166.777754] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 166.779351] arc_write_ready+0x3c7/0x7b0 [zfs] [ 166.780887] zio_ready+0x5b/0x510 [zfs] [ 166.782005] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 166.783267] zio_execute+0xbb/0x170 [zfs] [ 166.784786] taskq_thread+0x45c/0xbc0 [spl] [ 166.787202] ? try_to_wake_up+0x710/0x710 [ 166.789000] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 166.790939] ? taskq_thread_spawn+0x90/0x90 [spl] [ 166.792254] kthread+0x1d1/0x200 [ 166.793241] ? set_kthread_struct+0x70/0x70 [ 166.794268] ret_from_fork+0x1f/0x30 [ 166.795271] ---[ end trace 2e86c68e9c14ac29 ]--- [ 169.198276] ------------[ cut here ]------------ [ 169.200604] list_del corruption. next->prev should be ffff88b702d7a858, but was ffff88b818434c58 [ 169.206212] WARNING: CPU: 2 PID: 8002 at lib/list_debug.c:56 __list_del_entry_valid+0x122/0x140 [ 169.210272] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 169.237931] CPU: 2 PID: 8002 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 169.242386] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 169.245460] RIP: 0010:__list_del_entry_valid+0x122/0x140 [ 169.247441] Code: 0c 03 01 31 c0 48 83 05 eb 7c 0c 03 01 c3 48 c7 c7 30 e2 9e af 48 83 05 eb 7c 0c 03 01 e8 c2 48 8f ff 48 83 05 e6 7c 0c 03 01 <0f> 0b 48 83 05 e4 7c 0c 03 01 48 83 05 e4 7c 0c 03 01 31 c0 48 83 [ 169.251694] RSP: 0018:ffffa1be430977f0 EFLAGS: 00010206 [ 169.252799] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 169.254331] RDX: ffff88b84212e640 RSI: ffff88b84211e5a8 RDI: ffff88b84211e5a8 [ 169.256131] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 169.261038] R10: 0000000000000001 R11: ffffa1be430975e0 R12: ffffffffc18894f0 [ 169.263993] R13: ffff88b702d7a800 R14: 0000000000000000 R15: 0000000000600210 [ 169.267466] FS: 0000000000000000(0000) GS:ffff88b842100000(0000) knlGS:0000000000000000 [ 169.270483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 169.273753] CR2: 00007f5e1fd03180 CR3: 0000000038416005 CR4: 0000000000170ee0 [ 169.276282] Call Trace: [ 169.276888] ? show_regs.cold.9+0x22/0x2f [ 169.277898] ? __warn+0xc8/0x150 [ 169.278821] ? __list_del_entry_valid+0x122/0x140 [ 169.279858] ? report_bug+0x113/0x140 [ 169.280566] ? do_error_trap+0xb6/0x130 [ 169.281509] ? do_invalid_op+0x46/0x60 [ 169.282778] ? __list_del_entry_valid+0x122/0x140 [ 169.285466] ? invalid_op+0x14/0x20 [ 169.286748] ? __list_del_entry_valid+0x122/0x140 [ 169.288106] ldlm_namespace_move_to_active_locked+0x36/0xd0 [ptlrpc] [ 169.290099] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 169.291720] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 169.294865] do_shrink_slab+0x41/0x530 [ 169.296256] shrink_slab+0xff/0x4f0 [ 169.297301] shrink_node+0x321/0xc20 [ 169.298376] do_try_to_free_pages+0xfc/0x7a0 [ 169.299631] try_to_free_pages+0x144/0x330 [ 169.300981] __alloc_pages_slowpath+0x59b/0x1870 [ 169.303627] ? spl_kmem_cache_alloc+0x76/0x260 [spl] [ 169.305302] ? do_raw_spin_unlock+0x75/0x190 [ 169.306482] ? get_page_from_freelist+0x114/0x5d0 [ 169.307813] __alloc_pages_nodemask+0x4d3/0x550 [ 169.309252] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 169.312612] ? __mutex_init+0x67/0x80 [ 169.313800] abd_alloc+0x5d/0x100 [zfs] [ 169.315377] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 169.317244] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 169.319236] arc_write_ready+0x3c7/0x7b0 [zfs] [ 169.322911] zio_ready+0x5b/0x510 [zfs] [ 169.324632] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 169.326268] zio_execute+0xbb/0x170 [zfs] [ 169.328069] taskq_thread+0x45c/0xbc0 [spl] [ 169.330951] ? try_to_wake_up+0x710/0x710 [ 169.332804] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 169.335613] ? taskq_thread_spawn+0x90/0x90 [spl] [ 169.337585] kthread+0x1d1/0x200 [ 169.340040] ? set_kthread_struct+0x70/0x70 [ 169.341681] ret_from_fork+0x1f/0x30 [ 169.342683] ---[ end trace 2e86c68e9c14ac2a ]--- [ 169.344229] ------------[ cut here ]------------ [ 169.346116] list_add double add: new=ffff88b702d7a858, prev=ffff88b702d7a858, next=ffffffffc18894f0. [ 169.350621] WARNING: CPU: 2 PID: 8002 at lib/list_debug.c:31 __list_add_valid+0xd7/0x100 [ 169.352154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 169.365825] CPU: 2 PID: 8002 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 169.367974] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 169.372563] RIP: 0010:__list_add_valid+0xd7/0x100 [ 169.374150] Code: 7b 0c 03 01 c3 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 10 e3 9e af 48 83 05 4e 7b 0c 03 01 e8 cd 47 8f ff 48 83 05 49 7b 0c 03 01 <0f> 0b 48 83 05 47 7b 0c 03 01 48 83 05 47 7b 0c 03 01 31 c0 48 83 [ 169.380721] RSP: 0018:ffffa1be430977f0 EFLAGS: 00010202 [ 169.382505] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 169.385370] RDX: ffff88b84212e640 RSI: ffff88b84211e5a8 RDI: ffff88b84211e5a8 [ 169.387826] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 169.391467] R10: 0000000000000001 R11: ffffa1be430975e0 R12: ffffffffc18894f0 [ 169.393414] R13: ffff88b702d7a858 R14: 0000000000000000 R15: 0000000000600210 [ 169.395269] FS: 0000000000000000(0000) GS:ffff88b842100000(0000) knlGS:0000000000000000 [ 169.398366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 169.402268] CR2: 00007f5e1fd03180 CR3: 0000000038416005 CR4: 0000000000170ee0 [ 169.404598] Call Trace: [ 169.405574] ? show_regs.cold.9+0x22/0x2f [ 169.407241] ? __warn+0xc8/0x150 [ 169.408247] ? __list_add_valid+0xd7/0x100 [ 169.409856] ? report_bug+0x113/0x140 [ 169.413049] ? do_error_trap+0xb6/0x130 [ 169.414512] ? do_invalid_op+0x46/0x60 [ 169.415793] ? __list_add_valid+0xd7/0x100 [ 169.416829] ? invalid_op+0x14/0x20 [ 169.417747] ? __list_add_valid+0xd7/0x100 [ 169.419185] ldlm_namespace_move_to_active_locked+0x64/0xd0 [ptlrpc] [ 169.423459] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 169.425914] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 169.427844] do_shrink_slab+0x41/0x530 [ 169.429239] shrink_slab+0xff/0x4f0 [ 169.430858] shrink_node+0x321/0xc20 [ 169.433135] do_try_to_free_pages+0xfc/0x7a0 [ 169.435393] try_to_free_pages+0x144/0x330 [ 169.437092] __alloc_pages_slowpath+0x59b/0x1870 [ 169.438891] ? spl_kmem_cache_alloc+0x76/0x260 [spl] [ 169.442390] ? do_raw_spin_unlock+0x75/0x190 [ 169.444076] ? get_page_from_freelist+0x114/0x5d0 [ 169.446238] __alloc_pages_nodemask+0x4d3/0x550 [ 169.447448] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 169.449510] ? __mutex_init+0x67/0x80 [ 169.451036] abd_alloc+0x5d/0x100 [zfs] [ 169.454712] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 169.458139] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 169.460292] arc_write_ready+0x3c7/0x7b0 [zfs] [ 169.462688] zio_ready+0x5b/0x510 [zfs] [ 169.466455] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 169.468373] zio_execute+0xbb/0x170 [zfs] [ 169.469892] taskq_thread+0x45c/0xbc0 [spl] [ 169.470918] ? try_to_wake_up+0x710/0x710 [ 169.472090] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 169.474836] ? taskq_thread_spawn+0x90/0x90 [spl] [ 169.477489] kthread+0x1d1/0x200 [ 169.478543] ? set_kthread_struct+0x70/0x70 [ 169.479636] ret_from_fork+0x1f/0x30 [ 169.480515] ---[ end trace 2e86c68e9c14ac2b ]--- [ 176.559287] ------------[ cut here ]------------ [ 176.559287] ------------[ cut here ]------------ [ 176.559321] list_del corruption. prev->next should be ffff88b702d7a858, but was ffffffffc18894f0 [ 176.561725] list_del corruption. next->prev should be ffff88b702d7a858, but was ffff88b818434c58 [ 176.561767] WARNING: CPU: 1 PID: 7186 at lib/list_debug.c:56 __list_del_entry_valid+0x122/0x140 [ 176.563867] WARNING: CPU: 2 PID: 7234 at lib/list_debug.c:53 __list_del_entry_valid+0xe9/0x140 [ 176.569334] Modules linked in: [ 176.572825] Modules linked in: lustre(O) [ 176.575208] lustre(O) [ 176.578478] osp(O) [ 176.580071] osp(O) [ 176.581892] ofd(O) [ 176.582810] ofd(O) [ 176.583276] lod(O) [ 176.584049] lod(O) [ 176.584529] mdt(O) [ 176.585364] mdt(O) [ 176.586119] mdd(O) [ 176.587112] mdd(O) [ 176.587931] mgs(O) [ 176.590153] mgs(O) [ 176.592216] osd_zfs(O) [ 176.593405] osd_zfs(O) [ 176.594303] lquota(O) [ 176.595444] lquota(O) [ 176.596322] lfsck(O) [ 176.597151] lfsck(O) [ 176.598802] obdecho(O) [ 176.600247] obdecho(O) [ 176.601082] mgc(O) [ 176.603118] mgc(O) [ 176.605979] mdc(O) [ 176.608936] mdc(O) [ 176.609772] lov(O) [ 176.610353] lov(O) [ 176.611726] osc(O) [ 176.612496] osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [ 176.613331] lmv(O) [ 176.617027] sha512_generic [ 176.617971] fid(O) [ 176.639021] [ 176.641139] fld(O) [ 176.642603] CPU: 1 PID: 7186 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 176.643475] ptlrpc_gss(O) [ 176.645118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 176.646083] ptlrpc(O) [ 176.653467] RIP: 0010:__list_del_entry_valid+0x122/0x140 [ 176.654443] obdclass(O) [ 176.661104] Code: 0c 03 01 31 c0 48 83 05 eb 7c 0c 03 01 c3 48 c7 c7 30 e2 9e af 48 83 05 eb 7c 0c 03 01 e8 c2 48 8f ff 48 83 05 e6 7c 0c 03 01 <0f> 0b 48 83 05 e4 7c 0c 03 01 48 83 05 e4 7c 0c 03 01 31 c0 48 83 [ 176.662499] ksocklnd(O) lnet(O) [ 176.666464] RSP: 0018:ffffa1be437477f0 EFLAGS: 00010202 [ 176.668152] libcfs(O) [ 176.677858] [ 176.680535] zfs(O) [ 176.683334] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000001 [ 176.685381] spl(O) [ 176.686597] RDX: 0000000000000001 RSI: 0000000000000002 RDI: ffff88b84209e5a0 [ 176.688957] rpcsec_gss_krb5 [ 176.693133] RBP: ffff88b702d7a858 R08: ffffffffafd05780 R09: 0000000000000030 [ 176.694144] auth_rpcgss [ 176.700534] R10: ffffa1be4366dd4a R11: ffffffffb0f8ef3c R12: ffffffffc18894f0 [ 176.702055] nfsv4 [ 176.705861] R13: ffff88b702d7a800 R14: 0000000000000000 R15: 0000000000600210 [ 176.709432] dns_resolver [ 176.711960] FS: 0000000000000000(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 176.713547] intel_rapl_msr [ 176.718431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 176.720603] intel_rapl_common [ 176.724656] CR2: 00007fa524291020 CR3: 0000000038416003 CR4: 0000000000170ee0 [ 176.725952] sb_edac [ 176.730089] Call Trace: [ 176.731346] rapl [ 176.733933] ? show_regs.cold.9+0x22/0x2f [ 176.734667] pcspkr [ 176.735578] ? __warn+0xc8/0x150 [ 176.736286] i2c_piix4 [ 176.737987] ? __list_del_entry_valid+0x122/0x140 [ 176.738895] squashfs [ 176.740277] ? report_bug+0x113/0x140 [ 176.741205] crct10dif_pclmul [ 176.744632] ? do_error_trap+0xb6/0x130 [ 176.745114] crc32_pclmul [ 176.749040] ? do_invalid_op+0x46/0x60 [ 176.750291] crc32c_intel [ 176.751792] ? __list_del_entry_valid+0x122/0x140 [ 176.752662] ata_generic [ 176.754044] ? invalid_op+0x14/0x20 [ 176.754643] ghash_clmulni_intel [ 176.756501] ? __list_del_entry_valid+0x122/0x140 [ 176.757455] serio_raw [ 176.760815] ldlm_namespace_move_to_active_locked+0x36/0xd0 [ptlrpc] [ 176.762047] ata_piix [ 176.763800] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 176.764594] libata [ 176.767727] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 176.769914] dm_mirror [ 176.771894] do_shrink_slab+0x41/0x530 [ 176.772625] dm_region_hash [ 176.774478] shrink_slab+0xff/0x4f0 [ 176.775163] dm_log [ 176.776983] shrink_node+0x321/0xc20 [ 176.778167] dm_mod [ 176.780507] do_try_to_free_pages+0xfc/0x7a0 [ 176.781883] sha512_ssse3 [ 176.783375] try_to_free_pages+0x144/0x330 [ 176.784118] sha512_generic [ 176.785254] __alloc_pages_slowpath+0x59b/0x1870 [ 176.786103] [ 176.787911] ? _cond_resched+0x1d/0x50 [ 176.789581] CPU: 2 PID: 7234 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 176.792449] ? do_raw_spin_unlock+0x75/0x190 [ 176.793740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 176.795445] ? get_page_from_freelist+0x114/0x5d0 [ 176.799215] RIP: 0010:__list_del_entry_valid+0xe9/0x140 [ 176.801629] __alloc_pages_nodemask+0x4d3/0x550 [ 176.809992] Code: 83 05 02 7d 0c 03 01 c3 48 89 f2 48 89 fe 48 c7 c7 f0 e1 9e af 48 83 05 f4 7c 0c 03 01 e8 fb 48 8f ff 48 83 05 ef 7c 0c 03 01 <0f> 0b 48 83 05 ed 7c 0c 03 01 48 83 05 ed 7c 0c 03 01 31 c0 48 83 [ 176.811191] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 176.815626] RSP: 0018:ffffa1be4379f7f0 EFLAGS: 00010202 [ 176.817310] ? __mutex_init+0x67/0x80 [ 176.826787] [ 176.827924] abd_alloc+0x5d/0x100 [zfs] [ 176.829302] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 176.830164] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 176.830703] RDX: ffff88b84212e640 RSI: ffff88b84211e5a8 RDI: ffff88b84211e5a8 [ 176.831596] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 176.834091] RBP: ffff88b702d7a858 R08: 2d2d2d2d2d2d2d2d R09: 656820747563205b [ 176.837577] arc_write_ready+0x3c7/0x7b0 [zfs] [ 176.840184] R10: 000000002d2d2d2d R11: 000000002d2d2d2d R12: ffffffffc18894f0 [ 176.841594] zio_ready+0x5b/0x510 [zfs] [ 176.844195] R13: ffff88b702d7a800 R14: 0000000000000000 R15: 0000000000600210 [ 176.847140] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 176.849804] FS: 0000000000000000(0000) GS:ffff88b842100000(0000) knlGS:0000000000000000 [ 176.850825] zio_execute+0xbb/0x170 [zfs] [ 176.852681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 176.854097] taskq_thread+0x45c/0xbc0 [spl] [ 176.858771] CR2: 00007f5e1fd12c78 CR3: 0000000038416005 CR4: 0000000000170ee0 [ 176.860194] ? try_to_wake_up+0x710/0x710 [ 176.862041] Call Trace: [ 176.863211] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 176.867444] ? show_regs.cold.9+0x22/0x2f [ 176.868373] ? taskq_thread_spawn+0x90/0x90 [spl] [ 176.869415] ? __warn+0xc8/0x150 [ 176.871115] kthread+0x1d1/0x200 [ 176.872529] ? __list_del_entry_valid+0xe9/0x140 [ 176.876390] ? set_kthread_struct+0x70/0x70 [ 176.877362] ? report_bug+0x113/0x140 [ 176.878081] ret_from_fork+0x1f/0x30 [ 176.879478] ? do_error_trap+0xb6/0x130 [ 176.880389] ---[ end trace 2e86c68e9c14ac2c ]--- [ 176.881647] ? do_invalid_op+0x46/0x60 [ 176.883562] ------------[ cut here ]------------ [ 176.885585] ? __list_del_entry_valid+0xe9/0x140 [ 176.885600] ? invalid_op+0x14/0x20 [ 176.887344] list_add double add: new=ffff88b702d7a858, prev=ffff88b702d7a858, next=ffffffffc18894f0. [ 176.888599] ? __list_del_entry_valid+0xe9/0x140 [ 176.889901] WARNING: CPU: 1 PID: 7186 at lib/list_debug.c:31 __list_add_valid+0xd7/0x100 [ 176.891649] ldlm_namespace_move_to_active_locked+0x36/0xd0 [ptlrpc] [ 176.892698] Modules linked in: [ 176.898076] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 176.900072] lustre(O) [ 176.905574] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 176.908761] osp(O) [ 176.910070] do_shrink_slab+0x41/0x530 [ 176.913061] ofd(O) [ 176.914275] shrink_slab+0xff/0x4f0 [ 176.915906] lod(O) [ 176.916727] shrink_node+0x321/0xc20 [ 176.917878] mdt(O) [ 176.918674] do_try_to_free_pages+0xfc/0x7a0 [ 176.919472] mdd(O) [ 176.920274] try_to_free_pages+0x144/0x330 [ 176.921491] mgs(O) [ 176.924170] __alloc_pages_slowpath+0x59b/0x1870 [ 176.925350] osd_zfs(O) [ 176.926007] ? spl_kmem_cache_alloc+0x76/0x260 [spl] [ 176.927505] lquota(O) [ 176.928010] ? do_raw_spin_unlock+0x75/0x190 [ 176.928998] lfsck(O) [ 176.929671] ? get_page_from_freelist+0x114/0x5d0 [ 176.931150] obdecho(O) [ 176.931731] __alloc_pages_nodemask+0x4d3/0x550 [ 176.935572] mgc(O) [ 176.936627] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 176.938262] mdc(O) [ 176.938969] ? __mutex_init+0x67/0x80 [ 176.940743] lov(O) [ 176.941192] abd_alloc+0x5d/0x100 [zfs] [ 176.942897] osc(O) [ 176.943747] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 176.946480] lmv(O) [ 176.947228] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 176.948515] fid(O) [ 176.949426] arc_write_ready+0x3c7/0x7b0 [zfs] [ 176.950562] fld(O) [ 176.951084] zio_ready+0x5b/0x510 [zfs] [ 176.952868] ptlrpc_gss(O) [ 176.954127] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 176.957518] ptlrpc(O) [ 176.958243] zio_execute+0xbb/0x170 [zfs] [ 176.959638] obdclass(O) [ 176.960431] taskq_thread+0x45c/0xbc0 [spl] [ 176.961993] ksocklnd(O) [ 176.962963] ? try_to_wake_up+0x710/0x710 [ 176.964289] lnet(O) [ 176.965235] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 176.968197] libcfs(O) [ 176.969243] ? taskq_thread_spawn+0x90/0x90 [spl] [ 176.970455] zfs(O) [ 176.971182] kthread+0x1d1/0x200 [ 176.973080] spl(O) [ 176.975684] ? set_kthread_struct+0x70/0x70 [ 176.976961] rpcsec_gss_krb5 [ 176.977920] ret_from_fork+0x1f/0x30 [ 176.979095] auth_rpcgss [ 176.980001] ---[ end trace 2e86c68e9c14ac2d ]--- [ 176.981011] nfsv4 [ 176.982028] ------------[ cut here ]------------ [ 176.983121] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs [ 176.985278] list_add double add: new=ffff88b702d7a858, prev=ffff88b702d7a858, next=ffffffffc18894f0. [ 176.987648] crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 176.987676] CPU: 1 PID: 7186 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 176.988636] WARNING: CPU: 2 PID: 7234 at lib/list_debug.c:31 __list_add_valid+0xd7/0x100 [ 176.989827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 176.992783] Modules linked in: lustre(O) [ 176.996951] RIP: 0010:__list_add_valid+0xd7/0x100 [ 177.002671] osp(O) [ 177.005373] Code: 7b 0c 03 01 c3 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 10 e3 9e af 48 83 05 4e 7b 0c 03 01 e8 cd 47 8f ff 48 83 05 49 7b 0c 03 01 <0f> 0b 48 83 05 47 7b 0c 03 01 48 83 05 47 7b 0c 03 01 31 c0 48 83 [ 177.007197] ofd(O) [ 177.009265] RSP: 0018:ffffa1be437477f0 EFLAGS: 00010206 [ 177.011366] lod(O) [ 177.012746] [ 177.013234] mdt(O) [ 177.017486] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 177.018921] mdd(O) [ 177.020481] RDX: ffff88b8420ae640 RSI: ffff88b84209e5a8 RDI: ffff88b84209e5a8 [ 177.021009] mgs(O) [ 177.021329] RBP: ffff88b702d7a858 R08: 2d2d2d2d2d2d2d2d R09: 656820747563205b [ 177.021816] osd_zfs(O) [ 177.023459] R10: 000000002d2d2d2d R11: 000000002d2d2d2d R12: ffffffffc18894f0 [ 177.023986] lquota(O) [ 177.025703] R13: ffff88b702d7a858 R14: 0000000000000000 R15: 0000000000600210 [ 177.026215] lfsck(O) [ 177.028562] FS: 0000000000000000(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 177.029654] obdecho(O) [ 177.031268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 177.031858] mgc(O) [ 177.033492] CR2: 00007fa524291020 CR3: 0000000038416003 CR4: 0000000000170ee0 [ 177.034063] mdc(O) [ 177.035880] Call Trace: [ 177.036445] lov(O) [ 177.037923] ? show_regs.cold.9+0x22/0x2f [ 177.038447] osc(O) [ 177.040106] ? __warn+0xc8/0x150 [ 177.040607] lmv(O) [ 177.041173] ? __list_add_valid+0xd7/0x100 [ 177.041700] fid(O) [ 177.042646] ? report_bug+0x113/0x140 [ 177.043161] fld(O) [ 177.043917] ? do_error_trap+0xb6/0x130 [ 177.044414] ptlrpc_gss(O) [ 177.045371] ? do_invalid_op+0x46/0x60 [ 177.045876] ptlrpc(O) [ 177.046770] ? __list_add_valid+0xd7/0x100 [ 177.047284] obdclass(O) [ 177.048196] ? invalid_op+0x14/0x20 [ 177.048886] ksocklnd(O) [ 177.049763] ? __list_add_valid+0xd7/0x100 [ 177.050323] lnet(O) [ 177.051262] ldlm_namespace_move_to_active_locked+0x64/0xd0 [ptlrpc] [ 177.051972] libcfs(O) [ 177.052803] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 177.053515] zfs(O) [ 177.054470] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 177.055124] spl(O) [ 177.056578] do_shrink_slab+0x41/0x530 [ 177.057205] rpcsec_gss_krb5 [ 177.058722] shrink_slab+0xff/0x4f0 [ 177.059332] auth_rpcgss [ 177.063635] shrink_node+0x321/0xc20 [ 177.064426] nfsv4 [ 177.066017] do_try_to_free_pages+0xfc/0x7a0 [ 177.067415] dns_resolver [ 177.068869] try_to_free_pages+0x144/0x330 [ 177.071458] intel_rapl_msr [ 177.072780] __alloc_pages_slowpath+0x59b/0x1870 [ 177.073537] intel_rapl_common [ 177.074571] ? _cond_resched+0x1d/0x50 [ 177.075393] sb_edac [ 177.076679] ? do_raw_spin_unlock+0x75/0x190 [ 177.077990] rapl [ 177.081255] ? get_page_from_freelist+0x114/0x5d0 [ 177.082425] pcspkr [ 177.083944] __alloc_pages_nodemask+0x4d3/0x550 [ 177.084847] i2c_piix4 [ 177.085826] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 177.087650] squashfs [ 177.088503] ? __mutex_init+0x67/0x80 [ 177.092240] crct10dif_pclmul [ 177.092953] abd_alloc+0x5d/0x100 [zfs] [ 177.094637] crc32_pclmul [ 177.095541] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 177.096940] crc32c_intel [ 177.098073] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 177.099435] ata_generic [ 177.100408] arc_write_ready+0x3c7/0x7b0 [zfs] [ 177.102354] ghash_clmulni_intel [ 177.103292] zio_ready+0x5b/0x510 [zfs] [ 177.104891] serio_raw [ 177.105906] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 177.107217] ata_piix [ 177.108241] zio_execute+0xbb/0x170 [zfs] [ 177.109246] libata [ 177.109854] taskq_thread+0x45c/0xbc0 [spl] [ 177.111112] dm_mirror [ 177.111978] ? try_to_wake_up+0x710/0x710 [ 177.112896] dm_region_hash [ 177.113711] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 177.114778] dm_log [ 177.115718] ? taskq_thread_spawn+0x90/0x90 [spl] [ 177.116662] dm_mod [ 177.117714] kthread+0x1d1/0x200 [ 177.119095] sha512_ssse3 [ 177.119944] ? set_kthread_struct+0x70/0x70 [ 177.121008] sha512_generic [ 177.121797] ret_from_fork+0x1f/0x30 [ 177.122943] [ 177.123607] ---[ end trace 2e86c68e9c14ac2e ]--- [ 177.124998] CPU: 2 PID: 7234 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 177.132236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 177.136128] RIP: 0010:__list_add_valid+0xd7/0x100 [ 177.138360] Code: 7b 0c 03 01 c3 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 10 e3 9e af 48 83 05 4e 7b 0c 03 01 e8 cd 47 8f ff 48 83 05 49 7b 0c 03 01 <0f> 0b 48 83 05 47 7b 0c 03 01 48 83 05 47 7b 0c 03 01 31 c0 48 83 [ 177.145730] RSP: 0018:ffffa1be4379f7f0 EFLAGS: 00010202 [ 177.148134] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 177.151498] RDX: ffff88b84212e640 RSI: ffff88b84211e5a8 RDI: ffff88b84211e5a8 [ 177.158613] RBP: ffff88b702d7a858 R08: ffffffffb0f80066 R09: 0000000000000030 [ 177.165396] R10: 0000000034767366 R11: 0000000073666e20 R12: ffffffffc18894f0 [ 177.170747] R13: ffff88b702d7a858 R14: 0000000000000000 R15: 0000000000600210 [ 177.173841] FS: 0000000000000000(0000) GS:ffff88b842100000(0000) knlGS:0000000000000000 [ 177.176950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 177.181455] CR2: 00007f5e1fd12c78 CR3: 0000000038416005 CR4: 0000000000170ee0 [ 177.183714] Call Trace: [ 177.184319] ? show_regs.cold.9+0x22/0x2f [ 177.185339] ? __warn+0xc8/0x150 [ 177.186601] ? __list_add_valid+0xd7/0x100 [ 177.187964] ? report_bug+0x113/0x140 [ 177.189031] ? do_error_trap+0xb6/0x130 [ 177.189749] ? do_invalid_op+0x46/0x60 [ 177.190565] ? __list_add_valid+0xd7/0x100 [ 177.192116] ? invalid_op+0x14/0x20 [ 177.193125] ? __list_add_valid+0xd7/0x100 [ 177.193851] ldlm_namespace_move_to_active_locked+0x64/0xd0 [ptlrpc] [ 177.195214] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 177.196194] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 177.197235] do_shrink_slab+0x41/0x530 [ 177.197976] shrink_slab+0xff/0x4f0 [ 177.198700] shrink_node+0x321/0xc20 [ 177.199452] do_try_to_free_pages+0xfc/0x7a0 [ 177.201113] try_to_free_pages+0x144/0x330 [ 177.201986] __alloc_pages_slowpath+0x59b/0x1870 [ 177.203024] ? spl_kmem_cache_alloc+0x76/0x260 [spl] [ 177.204257] ? do_raw_spin_unlock+0x75/0x190 [ 177.205152] ? get_page_from_freelist+0x114/0x5d0 [ 177.206831] __alloc_pages_nodemask+0x4d3/0x550 [ 177.207881] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 177.210067] ? __mutex_init+0x67/0x80 [ 177.210899] abd_alloc+0x5d/0x100 [zfs] [ 177.212319] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 177.213660] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 177.215329] arc_write_ready+0x3c7/0x7b0 [zfs] [ 177.217168] zio_ready+0x5b/0x510 [zfs] [ 177.218484] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 177.219970] zio_execute+0xbb/0x170 [zfs] [ 177.221668] taskq_thread+0x45c/0xbc0 [spl] [ 177.223246] ? try_to_wake_up+0x710/0x710 [ 177.224341] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 177.226529] ? taskq_thread_spawn+0x90/0x90 [spl] [ 177.227961] kthread+0x1d1/0x200 [ 177.229091] ? set_kthread_struct+0x70/0x70 [ 177.230399] ret_from_fork+0x1f/0x30 [ 177.231339] ---[ end trace 2e86c68e9c14ac2f ]--- [ 185.110806] ------------[ cut here ]------------ [ 185.113478] list_add double add: new=ffff88b702d7a858, prev=ffff88b702d7a858, next=ffffffffc18894f0. [ 185.117091] WARNING: CPU: 1 PID: 8002 at lib/list_debug.c:31 __list_add_valid+0xd7/0x100 [ 185.121677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 185.145016] CPU: 1 PID: 8002 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 185.150198] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 185.153058] RIP: 0010:__list_add_valid+0xd7/0x100 [ 185.154465] Code: 7b 0c 03 01 c3 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 10 e3 9e af 48 83 05 4e 7b 0c 03 01 e8 cd 47 8f ff 48 83 05 49 7b 0c 03 01 <0f> 0b 48 83 05 47 7b 0c 03 01 48 83 05 47 7b 0c 03 01 31 c0 48 83 [ 185.160797] RSP: 0018:ffffa1be430977f0 EFLAGS: 00010206 [ 185.162027] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 185.163232] RDX: ffff88b8420ae640 RSI: ffff88b84209e5a8 RDI: ffff88b84209e5a8 [ 185.165947] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 185.168573] R10: 0000000000000001 R11: ffffa1be430975e0 R12: ffffffffc18894f0 [ 185.170982] R13: ffff88b702d7a858 R14: 0000000000000000 R15: 0000000000600210 [ 185.172630] FS: 0000000000000000(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 185.175803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 185.177183] CR2: 00007fa51f34c000 CR3: 0000000038416003 CR4: 0000000000170ee0 [ 185.179757] Call Trace: [ 185.181377] ? show_regs.cold.9+0x22/0x2f [ 185.183389] ? __warn+0xc8/0x150 [ 185.184510] ? __list_add_valid+0xd7/0x100 [ 185.185536] ? report_bug+0x113/0x140 [ 185.186140] ? do_error_trap+0xb6/0x130 [ 185.186755] ? do_invalid_op+0x46/0x60 [ 185.187478] ? __list_add_valid+0xd7/0x100 [ 185.188321] ? invalid_op+0x14/0x20 [ 185.189043] ? __list_add_valid+0xd7/0x100 [ 185.190118] ldlm_namespace_move_to_active_locked+0x64/0xd0 [ptlrpc] [ 185.194196] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 185.195827] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 185.197485] do_shrink_slab+0x41/0x530 [ 185.198602] shrink_slab+0xff/0x4f0 [ 185.199391] shrink_node+0x321/0xc20 [ 185.200706] do_try_to_free_pages+0xfc/0x7a0 [ 185.202816] try_to_free_pages+0x144/0x330 [ 185.203669] __alloc_pages_slowpath+0x59b/0x1870 [ 185.204464] ? _cond_resched+0x1d/0x50 [ 185.205090] ? do_raw_spin_unlock+0x75/0x190 [ 185.205758] ? get_page_from_freelist+0x114/0x5d0 [ 185.206732] __alloc_pages_nodemask+0x4d3/0x550 [ 185.207725] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 185.208990] ? __mutex_init+0x67/0x80 [ 185.211772] abd_alloc+0x5d/0x100 [zfs] [ 185.212908] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 185.214300] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 185.215813] arc_write_ready+0x3c7/0x7b0 [zfs] [ 185.217703] zio_ready+0x5b/0x510 [zfs] [ 185.220711] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 185.221936] zio_execute+0xbb/0x170 [zfs] [ 185.222983] taskq_thread+0x45c/0xbc0 [spl] [ 185.223768] ? try_to_wake_up+0x710/0x710 [ 185.225068] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 185.229135] ? taskq_thread_spawn+0x90/0x90 [spl] [ 185.230517] kthread+0x1d1/0x200 [ 185.231353] ? set_kthread_struct+0x70/0x70 [ 185.232680] ret_from_fork+0x1f/0x30 [ 185.233764] ---[ end trace 2e86c68e9c14ac30 ]--- [ 190.016188] ------------[ cut here ]------------ [ 190.020031] list_add double add: new=ffff88b702d7a858, prev=ffff88b702d7a858, next=ffffffffc18894f0. [ 190.025929] WARNING: CPU: 2 PID: 7234 at lib/list_debug.c:31 __list_add_valid+0xd7/0x100 [ 190.029896] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 190.051996] CPU: 2 PID: 7234 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 190.056710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 190.062021] RIP: 0010:__list_add_valid+0xd7/0x100 [ 190.063797] Code: 7b 0c 03 01 c3 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 10 e3 9e af 48 83 05 4e 7b 0c 03 01 e8 cd 47 8f ff 48 83 05 49 7b 0c 03 01 <0f> 0b 48 83 05 47 7b 0c 03 01 48 83 05 47 7b 0c 03 01 31 c0 48 83 [ 190.068186] RSP: 0018:ffffa1be4379f7f0 EFLAGS: 00010206 [ 190.069307] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 190.070955] RDX: ffff88b84212e640 RSI: ffff88b84211e5a8 RDI: ffff88b84211e5a8 [ 190.074337] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 190.076819] R10: 0000000000000001 R11: ffffa1be4379f5e0 R12: ffffffffc18894f0 [ 190.079327] R13: ffff88b702d7a858 R14: 0000000000000000 R15: 0000000000600210 [ 190.082491] FS: 0000000000000000(0000) GS:ffff88b842100000(0000) knlGS:0000000000000000 [ 190.086148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 190.088277] CR2: 00007f5e1fcb0000 CR3: 0000000038416005 CR4: 0000000000170ee0 [ 190.090976] Call Trace: [ 190.092076] ? show_regs.cold.9+0x22/0x2f [ 190.093834] ? __warn+0xc8/0x150 [ 190.094682] ? __list_add_valid+0xd7/0x100 [ 190.096585] ? report_bug+0x113/0x140 [ 190.098291] ? do_error_trap+0xb6/0x130 [ 190.101906] ? do_invalid_op+0x46/0x60 [ 190.103695] ? __list_add_valid+0xd7/0x100 [ 190.106183] ? invalid_op+0x14/0x20 [ 190.107718] ? __list_add_valid+0xd7/0x100 [ 190.111276] ldlm_namespace_move_to_active_locked+0x64/0xd0 [ptlrpc] [ 190.113811] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 190.115968] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 190.118622] do_shrink_slab+0x41/0x530 [ 190.120090] shrink_slab+0xff/0x4f0 [ 190.122165] shrink_node+0x321/0xc20 [ 190.124662] do_try_to_free_pages+0xfc/0x7a0 [ 190.126459] try_to_free_pages+0x144/0x330 [ 190.128203] __alloc_pages_slowpath+0x59b/0x1870 [ 190.130567] ? spl_kmem_cache_alloc+0x76/0x260 [spl] [ 190.133165] ? do_raw_spin_unlock+0x75/0x190 [ 190.136136] ? get_page_from_freelist+0x114/0x5d0 [ 190.137736] __alloc_pages_nodemask+0x4d3/0x550 [ 190.139457] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 190.141145] ? __mutex_init+0x67/0x80 [ 190.141906] abd_alloc+0x5d/0x100 [zfs] [ 190.143767] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 190.146137] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 190.147982] arc_write_ready+0x3c7/0x7b0 [zfs] [ 190.149398] zio_ready+0x5b/0x510 [zfs] [ 190.150535] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 190.151622] zio_execute+0xbb/0x170 [zfs] [ 190.152814] taskq_thread+0x45c/0xbc0 [spl] [ 190.153846] ? try_to_wake_up+0x710/0x710 [ 190.155094] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 190.156919] ? taskq_thread_spawn+0x90/0x90 [spl] [ 190.160341] kthread+0x1d1/0x200 [ 190.161142] ? set_kthread_struct+0x70/0x70 [ 190.161916] ret_from_fork+0x1f/0x30 [ 190.162792] ---[ end trace 2e86c68e9c14ac31 ]--- [ 198.473738] ------------[ cut here ]------------ [ 198.475216] list_del corruption. prev->next should be ffff88b702d7a858, but was ffffffffc18894f0 [ 198.479441] WARNING: CPU: 1 PID: 8002 at lib/list_debug.c:53 __list_del_entry_valid+0xe9/0x140 [ 198.484091] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 198.506227] CPU: 1 PID: 8002 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 198.509433] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 198.513272] RIP: 0010:__list_del_entry_valid+0xe9/0x140 [ 198.514907] Code: 83 05 02 7d 0c 03 01 c3 48 89 f2 48 89 fe 48 c7 c7 f0 e1 9e af 48 83 05 f4 7c 0c 03 01 e8 fb 48 8f ff 48 83 05 ef 7c 0c 03 01 <0f> 0b 48 83 05 ed 7c 0c 03 01 48 83 05 ed 7c 0c 03 01 31 c0 48 83 [ 198.520768] RSP: 0018:ffffa1be430977f0 EFLAGS: 00010202 [ 198.522424] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 198.524484] RDX: ffff88b8420ae640 RSI: ffff88b84209e5a8 RDI: ffff88b84209e5a8 [ 198.526745] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 198.530988] R10: 0000000000000001 R11: ffffa1be430975e0 R12: ffffffffc18894f0 [ 198.533696] R13: ffff88b702d7a800 R14: 0000000000000000 R15: 0000000000600210 [ 198.536380] FS: 0000000000000000(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 198.540217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 198.541304] CR2: 00007fa51f361000 CR3: 0000000038416003 CR4: 0000000000170ee0 [ 198.542862] Call Trace: [ 198.543765] ? show_regs.cold.9+0x22/0x2f [ 198.544824] ? __warn+0xc8/0x150 [ 198.546599] ? __list_del_entry_valid+0xe9/0x140 [ 198.548497] ? report_bug+0x113/0x140 [ 198.549238] ? do_error_trap+0xb6/0x130 [ 198.550198] ? do_invalid_op+0x46/0x60 [ 198.551357] ? __list_del_entry_valid+0xe9/0x140 [ 198.552469] ? invalid_op+0x14/0x20 [ 198.553835] ? __list_del_entry_valid+0xe9/0x140 [ 198.556703] ldlm_namespace_move_to_active_locked+0x36/0xd0 [ptlrpc] [ 198.558563] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 198.559601] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 198.560701] do_shrink_slab+0x41/0x530 [ 198.561805] shrink_slab+0xff/0x4f0 [ 198.562843] shrink_node+0x321/0xc20 [ 198.564947] do_try_to_free_pages+0xfc/0x7a0 [ 198.566205] try_to_free_pages+0x144/0x330 [ 198.567307] __alloc_pages_slowpath+0x59b/0x1870 [ 198.568327] ? _cond_resched+0x1d/0x50 [ 198.569062] ? do_raw_spin_unlock+0x75/0x190 [ 198.569787] ? get_page_from_freelist+0x114/0x5d0 [ 198.570709] __alloc_pages_nodemask+0x4d3/0x550 [ 198.571604] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 198.572983] ? __mutex_init+0x67/0x80 [ 198.574565] abd_alloc+0x5d/0x100 [zfs] [ 198.575770] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 198.577426] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 198.579647] arc_write_ready+0x3c7/0x7b0 [zfs] [ 198.581821] zio_ready+0x5b/0x510 [zfs] [ 198.585161] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 198.587320] zio_execute+0xbb/0x170 [zfs] [ 198.589379] taskq_thread+0x45c/0xbc0 [spl] [ 198.591088] ? try_to_wake_up+0x710/0x710 [ 198.592804] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 198.597569] ? taskq_thread_spawn+0x90/0x90 [spl] [ 198.599268] kthread+0x1d1/0x200 [ 198.600843] ? set_kthread_struct+0x70/0x70 [ 198.603952] ret_from_fork+0x1f/0x30 [ 198.606818] ---[ end trace 2e86c68e9c14ac32 ]--- [ 198.609833] ------------[ cut here ]------------ [ 198.611022] list_add double add: new=ffff88b702d7a858, prev=ffff88b702d7a858, next=ffffffffc18894f0. [ 198.613223] WARNING: CPU: 1 PID: 8002 at lib/list_debug.c:31 __list_add_valid+0xd7/0x100 [ 198.615844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 198.631255] CPU: 1 PID: 8002 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 198.633323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 198.635984] RIP: 0010:__list_add_valid+0xd7/0x100 [ 198.639323] Code: 7b 0c 03 01 c3 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 10 e3 9e af 48 83 05 4e 7b 0c 03 01 e8 cd 47 8f ff 48 83 05 49 7b 0c 03 01 <0f> 0b 48 83 05 47 7b 0c 03 01 48 83 05 47 7b 0c 03 01 31 c0 48 83 [ 198.644757] RSP: 0018:ffffa1be430977f0 EFLAGS: 00010202 [ 198.647838] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 198.651283] RDX: ffff88b8420ae640 RSI: ffff88b84209e5a8 RDI: ffff88b84209e5a8 [ 198.654458] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 198.658828] R10: 0000000000000001 R11: ffffa1be430975e0 R12: ffffffffc18894f0 [ 198.661794] R13: ffff88b702d7a858 R14: 0000000000000000 R15: 0000000000600210 [ 198.666415] FS: 0000000000000000(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 198.670014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 198.672398] CR2: 00007fa51f361000 CR3: 0000000038416003 CR4: 0000000000170ee0 [ 198.675401] Call Trace: [ 198.676465] ? show_regs.cold.9+0x22/0x2f [ 198.678165] ? __warn+0xc8/0x150 [ 198.679518] ? __list_add_valid+0xd7/0x100 [ 198.681239] ? report_bug+0x113/0x140 [ 198.682796] ? do_error_trap+0xb6/0x130 [ 198.684423] ? do_invalid_op+0x46/0x60 [ 198.686020] ? __list_add_valid+0xd7/0x100 [ 198.687746] ? invalid_op+0x14/0x20 [ 198.689190] ? __list_add_valid+0xd7/0x100 [ 198.690930] ldlm_namespace_move_to_active_locked+0x64/0xd0 [ptlrpc] [ 198.693901] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 198.696158] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 198.698480] do_shrink_slab+0x41/0x530 [ 198.700125] shrink_slab+0xff/0x4f0 [ 198.701614] shrink_node+0x321/0xc20 [ 198.703159] do_try_to_free_pages+0xfc/0x7a0 [ 198.705006] try_to_free_pages+0x144/0x330 [ 198.708156] __alloc_pages_slowpath+0x59b/0x1870 [ 198.710358] ? _cond_resched+0x1d/0x50 [ 198.712005] ? do_raw_spin_unlock+0x75/0x190 [ 198.714718] ? get_page_from_freelist+0x114/0x5d0 [ 198.716235] __alloc_pages_nodemask+0x4d3/0x550 [ 198.717264] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 198.718573] ? __mutex_init+0x67/0x80 [ 198.719151] abd_alloc+0x5d/0x100 [zfs] [ 198.720326] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 198.721949] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 198.725473] arc_write_ready+0x3c7/0x7b0 [zfs] [ 198.727599] zio_ready+0x5b/0x510 [zfs] [ 198.729029] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 198.731913] zio_execute+0xbb/0x170 [zfs] [ 198.733209] taskq_thread+0x45c/0xbc0 [spl] [ 198.734480] ? try_to_wake_up+0x710/0x710 [ 198.735760] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 198.737370] ? taskq_thread_spawn+0x90/0x90 [spl] [ 198.738758] kthread+0x1d1/0x200 [ 198.739304] ? set_kthread_struct+0x70/0x70 [ 198.740010] ret_from_fork+0x1f/0x30 [ 198.740846] ---[ end trace 2e86c68e9c14ac33 ]--- [ 224.690567] ------------[ cut here ]------------ [ 224.692896] list_del corruption. next->prev should be ffff88b702d7a858, but was ffff88b818434c58 [ 224.697220] WARNING: CPU: 2 PID: 6323 at lib/list_debug.c:56 __list_del_entry_valid+0x122/0x140 [ 224.702800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 224.719346] CPU: 2 PID: 6323 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 224.722592] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 224.726221] RIP: 0010:__list_del_entry_valid+0x122/0x140 [ 224.727482] Code: 0c 03 01 31 c0 48 83 05 eb 7c 0c 03 01 c3 48 c7 c7 30 e2 9e af 48 83 05 eb 7c 0c 03 01 e8 c2 48 8f ff 48 83 05 e6 7c 0c 03 01 <0f> 0b 48 83 05 e4 7c 0c 03 01 48 83 05 e4 7c 0c 03 01 31 c0 48 83 [ 224.731436] RSP: 0018:ffffa1be4334b7f0 EFLAGS: 00010206 [ 224.732455] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 224.735062] RDX: ffff88b84212e640 RSI: ffff88b84211e5a8 RDI: ffff88b84211e5a8 [ 224.738427] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 224.740502] R10: 0000000000000001 R11: ffffa1be4334b5e0 R12: ffffffffc18894f0 [ 224.742265] R13: ffff88b702d7a800 R14: 0000000000000000 R15: 0000000000600210 [ 224.743928] FS: 0000000000000000(0000) GS:ffff88b842100000(0000) knlGS:0000000000000000 [ 224.747583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 224.748682] CR2: 00007fa51f36a008 CR3: 0000000038416001 CR4: 0000000000170ee0 [ 224.750833] Call Trace: [ 224.751514] ? show_regs.cold.9+0x22/0x2f [ 224.752512] ? __warn+0xc8/0x150 [ 224.753158] ? __list_del_entry_valid+0x122/0x140 [ 224.754251] ? report_bug+0x113/0x140 [ 224.756867] ? do_error_trap+0xb6/0x130 [ 224.758031] ? do_invalid_op+0x46/0x60 [ 224.759177] ? __list_del_entry_valid+0x122/0x140 [ 224.760611] ? invalid_op+0x14/0x20 [ 224.761639] ? __list_del_entry_valid+0x122/0x140 [ 224.763051] ldlm_namespace_move_to_active_locked+0x36/0xd0 [ptlrpc] [ 224.765436] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 224.768364] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 224.770153] do_shrink_slab+0x41/0x530 [ 224.771292] shrink_slab+0xff/0x4f0 [ 224.772366] shrink_node+0x321/0xc20 [ 224.773451] do_try_to_free_pages+0xfc/0x7a0 [ 224.774640] try_to_free_pages+0x144/0x330 [ 224.776012] __alloc_pages_slowpath+0x59b/0x1870 [ 224.778674] ? _cond_resched+0x1d/0x50 [ 224.780020] ? do_raw_spin_unlock+0x75/0x190 [ 224.781203] ? get_page_from_freelist+0x114/0x5d0 [ 224.782286] __alloc_pages_nodemask+0x4d3/0x550 [ 224.783691] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 224.785828] ? __mutex_init+0x67/0x80 [ 224.788053] abd_alloc+0x5d/0x100 [zfs] [ 224.789845] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 224.791437] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 224.793587] arc_write_ready+0x3c7/0x7b0 [zfs] [ 224.794962] zio_ready+0x5b/0x510 [zfs] [ 224.796203] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 224.799287] zio_execute+0xbb/0x170 [zfs] [ 224.801105] taskq_thread+0x45c/0xbc0 [spl] [ 224.802447] ? try_to_wake_up+0x710/0x710 [ 224.803583] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 224.806140] ? taskq_thread_spawn+0x90/0x90 [spl] [ 224.807593] kthread+0x1d1/0x200 [ 224.809169] ? set_kthread_struct+0x70/0x70 [ 224.810054] ret_from_fork+0x1f/0x30 [ 224.810981] ---[ end trace 2e86c68e9c14ac34 ]--- [ 224.812051] ------------[ cut here ]------------ [ 224.813065] list_add double add: new=ffff88b702d7a858, prev=ffff88b702d7a858, next=ffffffffc18894f0. [ 224.816512] WARNING: CPU: 2 PID: 6323 at lib/list_debug.c:31 __list_add_valid+0xd7/0x100 [ 224.820900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 224.840699] CPU: 2 PID: 6323 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 224.844271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 224.845936] RIP: 0010:__list_add_valid+0xd7/0x100 [ 224.846816] Code: 7b 0c 03 01 c3 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 10 e3 9e af 48 83 05 4e 7b 0c 03 01 e8 cd 47 8f ff 48 83 05 49 7b 0c 03 01 <0f> 0b 48 83 05 47 7b 0c 03 01 48 83 05 47 7b 0c 03 01 31 c0 48 83 [ 224.851222] RSP: 0018:ffffa1be4334b7f0 EFLAGS: 00010202 [ 224.853434] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 224.854787] RDX: ffff88b84212e640 RSI: ffff88b84211e5a8 RDI: ffff88b84211e5a8 [ 224.856381] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 224.857673] R10: 0000000000000001 R11: ffffa1be4334b5e0 R12: ffffffffc18894f0 [ 224.859104] R13: ffff88b702d7a858 R14: 0000000000000000 R15: 0000000000600210 [ 224.860698] FS: 0000000000000000(0000) GS:ffff88b842100000(0000) knlGS:0000000000000000 [ 224.862195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 224.863209] CR2: 00007fa51f36a008 CR3: 0000000038416001 CR4: 0000000000170ee0 [ 224.864501] Call Trace: [ 224.865027] ? show_regs.cold.9+0x22/0x2f [ 224.865728] ? __warn+0xc8/0x150 [ 224.866304] ? __list_add_valid+0xd7/0x100 [ 224.867172] ? report_bug+0x113/0x140 [ 224.868126] ? do_error_trap+0xb6/0x130 [ 224.868834] ? do_invalid_op+0x46/0x60 [ 224.869751] ? __list_add_valid+0xd7/0x100 [ 224.871342] ? invalid_op+0x14/0x20 [ 224.873374] ? __list_add_valid+0xd7/0x100 [ 224.874249] ldlm_namespace_move_to_active_locked+0x64/0xd0 [ptlrpc] [ 224.875594] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 224.876946] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 224.878489] do_shrink_slab+0x41/0x530 [ 224.879210] shrink_slab+0xff/0x4f0 [ 224.879815] shrink_node+0x321/0xc20 [ 224.880577] do_try_to_free_pages+0xfc/0x7a0 [ 224.882363] try_to_free_pages+0x144/0x330 [ 224.884452] __alloc_pages_slowpath+0x59b/0x1870 [ 224.887305] ? _cond_resched+0x1d/0x50 [ 224.889328] ? do_raw_spin_unlock+0x75/0x190 [ 224.890154] ? get_page_from_freelist+0x114/0x5d0 [ 224.891241] __alloc_pages_nodemask+0x4d3/0x550 [ 224.894106] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 224.896185] ? __mutex_init+0x67/0x80 [ 224.897281] abd_alloc+0x5d/0x100 [zfs] [ 224.898922] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 224.901421] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 224.905058] arc_write_ready+0x3c7/0x7b0 [zfs] [ 224.906954] zio_ready+0x5b/0x510 [zfs] [ 224.908216] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 224.909296] zio_execute+0xbb/0x170 [zfs] [ 224.911014] taskq_thread+0x45c/0xbc0 [spl] [ 224.912691] ? try_to_wake_up+0x710/0x710 [ 224.914139] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 224.916308] ? taskq_thread_spawn+0x90/0x90 [spl] [ 224.917186] kthread+0x1d1/0x200 [ 224.917890] ? set_kthread_struct+0x70/0x70 [ 224.918661] ret_from_fork+0x1f/0x30 [ 224.919303] ---[ end trace 2e86c68e9c14ac35 ]--- [ 236.568943] ------------[ cut here ]------------ [ 236.570382] list_del corruption. prev->next should be ffff88b702d7a858, but was ffffffffc18894f0 [ 236.572920] WARNING: CPU: 0 PID: 11291 at lib/list_debug.c:53 __list_del_entry_valid+0xe9/0x140 [ 236.577385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 236.602853] CPU: 0 PID: 11291 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 236.607101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 236.609821] RIP: 0010:__list_del_entry_valid+0xe9/0x140 [ 236.612022] Code: 83 05 02 7d 0c 03 01 c3 48 89 f2 48 89 fe 48 c7 c7 f0 e1 9e af 48 83 05 f4 7c 0c 03 01 e8 fb 48 8f ff 48 83 05 ef 7c 0c 03 01 <0f> 0b 48 83 05 ed 7c 0c 03 01 48 83 05 ed 7c 0c 03 01 31 c0 48 83 [ 236.616642] RSP: 0018:ffffa1be42ef37f0 EFLAGS: 00010206 [ 236.620371] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 236.622149] RDX: ffff88b84202e640 RSI: ffff88b84201e5a8 RDI: ffff88b84201e5a8 [ 236.624270] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 236.627881] R10: 0000000000000001 R11: ffffa1be42ef35e0 R12: ffffffffc18894f0 [ 236.629886] R13: ffff88b702d7a800 R14: 0000000000000000 R15: 0000000000600210 [ 236.632274] FS: 0000000000000000(0000) GS:ffff88b842000000(0000) knlGS:0000000000000000 [ 236.636193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 236.637543] CR2: 00007fa51f373000 CR3: 0000000038416001 CR4: 0000000000170ef0 [ 236.639142] Call Trace: [ 236.639708] ? show_regs.cold.9+0x22/0x2f [ 236.640759] ? __warn+0xc8/0x150 [ 236.642683] ? __list_del_entry_valid+0xe9/0x140 [ 236.643939] ? report_bug+0x113/0x140 [ 236.644794] ? do_error_trap+0xb6/0x130 [ 236.645686] ? do_invalid_op+0x46/0x60 [ 236.646578] ? __list_del_entry_valid+0xe9/0x140 [ 236.647641] ? invalid_op+0x14/0x20 [ 236.648445] ? __list_del_entry_valid+0xe9/0x140 [ 236.649620] ldlm_namespace_move_to_active_locked+0x36/0xd0 [ptlrpc] [ 236.652488] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 236.653944] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 236.655383] do_shrink_slab+0x41/0x530 [ 236.656250] shrink_slab+0xff/0x4f0 [ 236.657111] ? move_freepages_block+0x1f2/0x2b0 [ 236.658337] shrink_node+0x321/0xc20 [ 236.660600] do_try_to_free_pages+0xfc/0x7a0 [ 236.661563] try_to_free_pages+0x144/0x330 [ 236.662461] __alloc_pages_slowpath+0x59b/0x1870 [ 236.663884] ? prep_new_page+0x24a/0x260 [ 236.664693] ? get_page_from_freelist+0x114/0x5d0 [ 236.666069] __alloc_pages_nodemask+0x4d3/0x550 [ 236.667473] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 236.670813] ? __mutex_init+0x67/0x80 [ 236.672128] abd_alloc+0x5d/0x100 [zfs] [ 236.673775] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 236.675740] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 236.678251] arc_write_ready+0x3c7/0x7b0 [zfs] [ 236.679578] zio_ready+0x5b/0x510 [zfs] [ 236.680903] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 236.682089] zio_execute+0xbb/0x170 [zfs] [ 236.683443] taskq_thread+0x45c/0xbc0 [spl] [ 236.684411] ? try_to_wake_up+0x710/0x710 [ 236.685219] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 236.688557] ? taskq_thread_spawn+0x90/0x90 [spl] [ 236.689747] kthread+0x1d1/0x200 [ 236.690664] ? set_kthread_struct+0x70/0x70 [ 236.691717] ret_from_fork+0x1f/0x30 [ 236.692868] ---[ end trace 2e86c68e9c14ac36 ]--- [ 236.694830] ------------[ cut here ]------------ [ 236.698548] list_add double add: new=ffff88b702d7a858, prev=ffff88b702d7a858, next=ffffffffc18894f0. [ 236.701956] WARNING: CPU: 0 PID: 11291 at lib/list_debug.c:31 __list_add_valid+0xd7/0x100 [ 236.704501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 236.719009] CPU: 0 PID: 11291 Comm: z_wr_iss Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 236.723114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 236.729136] RIP: 0010:__list_add_valid+0xd7/0x100 [ 236.733172] Code: 7b 0c 03 01 c3 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 10 e3 9e af 48 83 05 4e 7b 0c 03 01 e8 cd 47 8f ff 48 83 05 49 7b 0c 03 01 <0f> 0b 48 83 05 47 7b 0c 03 01 48 83 05 47 7b 0c 03 01 31 c0 48 83 [ 236.739009] RSP: 0018:ffffa1be42ef37f0 EFLAGS: 00010206 [ 236.741362] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 236.744041] RDX: ffff88b84202e640 RSI: ffff88b84201e5a8 RDI: ffff88b84201e5a8 [ 236.745396] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 236.746559] R10: 0000000000000001 R11: ffffa1be42ef35e0 R12: ffffffffc18894f0 [ 236.748158] R13: ffff88b702d7a858 R14: 0000000000000000 R15: 0000000000600210 [ 236.751288] FS: 0000000000000000(0000) GS:ffff88b842000000(0000) knlGS:0000000000000000 [ 236.753317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 236.754784] CR2: 00007fa51f373000 CR3: 0000000038416001 CR4: 0000000000170ef0 [ 236.756616] Call Trace: [ 236.757394] ? show_regs.cold.9+0x22/0x2f [ 236.758130] ? __warn+0xc8/0x150 [ 236.758949] ? __list_add_valid+0xd7/0x100 [ 236.759714] ? report_bug+0x113/0x140 [ 236.760311] ? do_error_trap+0xb6/0x130 [ 236.761040] ? do_invalid_op+0x46/0x60 [ 236.761816] ? __list_add_valid+0xd7/0x100 [ 236.762653] ? invalid_op+0x14/0x20 [ 236.763251] ? __list_add_valid+0xd7/0x100 [ 236.763958] ldlm_namespace_move_to_active_locked+0x64/0xd0 [ptlrpc] [ 236.765533] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 236.766831] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 236.767846] do_shrink_slab+0x41/0x530 [ 236.768504] shrink_slab+0xff/0x4f0 [ 236.769110] ? move_freepages_block+0x1f2/0x2b0 [ 236.769945] shrink_node+0x321/0xc20 [ 236.770614] do_try_to_free_pages+0xfc/0x7a0 [ 236.771364] try_to_free_pages+0x144/0x330 [ 236.772240] __alloc_pages_slowpath+0x59b/0x1870 [ 236.773139] ? prep_new_page+0x24a/0x260 [ 236.773913] ? get_page_from_freelist+0x114/0x5d0 [ 236.774907] __alloc_pages_nodemask+0x4d3/0x550 [ 236.776003] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 236.777284] ? __mutex_init+0x67/0x80 [ 236.777922] abd_alloc+0x5d/0x100 [zfs] [ 236.779136] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 236.780495] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 236.781859] arc_write_ready+0x3c7/0x7b0 [zfs] [ 236.783027] zio_ready+0x5b/0x510 [zfs] [ 236.784004] ? tsd_hash_search.isra.3+0x94/0xd0 [spl] [ 236.784918] zio_execute+0xbb/0x170 [zfs] [ 236.786002] taskq_thread+0x45c/0xbc0 [spl] [ 236.786768] ? try_to_wake_up+0x710/0x710 [ 236.787495] ? zio_taskq_member.isra.12.constprop.13+0xa0/0xa0 [zfs] [ 236.789131] ? taskq_thread_spawn+0x90/0x90 [spl] [ 236.789981] kthread+0x1d1/0x200 [ 236.790791] ? set_kthread_struct+0x70/0x70 [ 236.791794] ret_from_fork+0x1f/0x30 [ 236.792471] ---[ end trace 2e86c68e9c14ac37 ]--- [ 256.207683] ------------[ cut here ]------------ [ 256.212185] list_del corruption. prev->next should be ffff88b702d7a858, but was ffffffffc18894f0 [ 256.214132] WARNING: CPU: 1 PID: 11570 at lib/list_debug.c:53 __list_del_entry_valid+0xe9/0x140 [ 256.215769] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 256.237961] CPU: 1 PID: 11570 Comm: ll_ost_io00_015 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 256.244261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 256.247073] RIP: 0010:__list_del_entry_valid+0xe9/0x140 [ 256.248971] Code: 83 05 02 7d 0c 03 01 c3 48 89 f2 48 89 fe 48 c7 c7 f0 e1 9e af 48 83 05 f4 7c 0c 03 01 e8 fb 48 8f ff 48 83 05 ef 7c 0c 03 01 <0f> 0b 48 83 05 ed 7c 0c 03 01 48 83 05 ed 7c 0c 03 01 31 c0 48 83 [ 256.257475] RSP: 0018:ffffa1be434631c0 EFLAGS: 00010202 [ 256.262122] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 256.266685] RDX: ffff88b8420ae640 RSI: ffff88b84209e5a8 RDI: ffff88b84209e5a8 [ 256.268281] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 256.270608] R10: 0000000000000001 R11: ffffa1be43462fb0 R12: ffffffffc18894f0 [ 256.273273] R13: ffff88b702d7a800 R14: 0000000000000000 R15: 0000000000600210 [ 256.277261] FS: 0000000000000000(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 256.278671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 256.281067] CR2: 00007f5e1c051000 CR3: 0000000038416003 CR4: 0000000000170ee0 [ 256.283215] Call Trace: [ 256.284595] ? show_regs.cold.9+0x22/0x2f [ 256.286250] ? __warn+0xc8/0x150 [ 256.290460] ? __list_del_entry_valid+0xe9/0x140 [ 256.292014] ? report_bug+0x113/0x140 [ 256.293652] ? do_error_trap+0xb6/0x130 [ 256.295322] ? do_invalid_op+0x46/0x60 [ 256.296957] ? __list_del_entry_valid+0xe9/0x140 [ 256.298913] ? invalid_op+0x14/0x20 [ 256.300447] ? __list_del_entry_valid+0xe9/0x140 [ 256.302480] ldlm_namespace_move_to_active_locked+0x36/0xd0 [ptlrpc] [ 256.305442] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 256.307736] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 256.310117] do_shrink_slab+0x41/0x530 [ 256.311811] shrink_slab+0xff/0x4f0 [ 256.313342] ? move_freepages_block+0x1f2/0x2b0 [ 256.315246] shrink_node+0x321/0xc20 [ 256.316741] do_try_to_free_pages+0xfc/0x7a0 [ 256.318510] try_to_free_pages+0x144/0x330 [ 256.320211] __alloc_pages_slowpath+0x59b/0x1870 [ 256.322140] ? blk_finish_plug+0x31/0x46 [ 256.323766] ? get_page_from_freelist+0x114/0x5d0 [ 256.325702] __alloc_pages_nodemask+0x4d3/0x550 [ 256.327583] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 256.329975] ? __mutex_init+0x67/0x80 [ 256.331523] abd_alloc+0x5d/0x100 [zfs] [ 256.333769] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 256.336464] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 256.338851] arc_read+0xbd2/0x2650 [zfs] [ 256.340988] ? dbuf_issue_final_prefetch+0x190/0x190 [zfs] [ 256.343812] ? zfetch_kstats_update+0x110/0x110 [zfs] [ 256.346434] dbuf_issue_final_prefetch+0xc7/0x190 [zfs] [ 256.349081] dbuf_prefetch_impl+0x8c1/0x9e0 [zfs] [ 256.351521] ? zfetch_kstats_update+0x110/0x110 [zfs] [ 256.354139] dmu_zfetch_run+0x1c0/0x3a0 [zfs] [ 256.356490] dmu_buf_hold_array_by_dnode+0x33b/0x770 [zfs] [ 256.359280] dmu_buf_hold_array_by_bonus+0x1c/0x30 [zfs] [ 256.362055] osd_bufs_get_read+0x1e4/0x950 [osd_zfs] [ 256.364172] osd_bufs_get+0x141/0x1e0 [osd_zfs] [ 256.366116] ofd_preprw_read.isra.15+0x2f1/0x1770 [ofd] [ 256.368354] ofd_preprw+0x5e3/0xcb0 [ofd] [ 256.370096] tgt_brw_read+0x769/0x26c0 [ptlrpc] [ 256.372366] ? sptlrpc_svc_alloc_rs+0x70/0x460 [ptlrpc] [ 256.374884] ? lustre_msg_add_version+0x29/0xd0 [ptlrpc] [ 256.377502] ? lustre_pack_reply_flags+0x72/0x250 [ptlrpc] [ 256.380216] ? lustre_pack_reply+0x15/0x20 [ptlrpc] [ 256.382613] ? req_capsule_server_pack+0x250/0x3d0 [ptlrpc] [ 256.385301] tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 256.387798] tgt_request_handle+0x351/0x1c00 [ptlrpc] [ 256.390297] ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 256.393135] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 256.395625] ptlrpc_main+0xd1e/0x1440 [ptlrpc] [ 256.397795] ? ptlrpc_wait_event+0x980/0x980 [ptlrpc] [ 256.400211] kthread+0x1d1/0x200 [ 256.401622] ? set_kthread_struct+0x70/0x70 [ 256.402788] ret_from_fork+0x1f/0x30 [ 256.403484] ---[ end trace 2e86c68e9c14ac38 ]--- [ 256.404502] ------------[ cut here ]------------ [ 256.405755] list_add double add: new=ffff88b702d7a858, prev=ffff88b702d7a858, next=ffffffffc18894f0. [ 256.407561] WARNING: CPU: 1 PID: 11570 at lib/list_debug.c:31 __list_add_valid+0xd7/0x100 [ 256.409324] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 256.420288] CPU: 1 PID: 11570 Comm: ll_ost_io00_015 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 256.422331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 256.423826] RIP: 0010:__list_add_valid+0xd7/0x100 [ 256.424820] Code: 7b 0c 03 01 c3 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 10 e3 9e af 48 83 05 4e 7b 0c 03 01 e8 cd 47 8f ff 48 83 05 49 7b 0c 03 01 <0f> 0b 48 83 05 47 7b 0c 03 01 48 83 05 47 7b 0c 03 01 31 c0 48 83 [ 256.428633] RSP: 0018:ffffa1be434631c0 EFLAGS: 00010206 [ 256.429754] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 256.431259] RDX: ffff88b8420ae640 RSI: ffff88b84209e5a8 RDI: ffff88b84209e5a8 [ 256.432733] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 256.433967] R10: 0000000000000001 R11: ffffa1be43462fb0 R12: ffffffffc18894f0 [ 256.435494] R13: ffff88b702d7a858 R14: 0000000000000000 R15: 0000000000600210 [ 256.436930] FS: 0000000000000000(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 256.438641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 256.439558] CR2: 00007f5e1c051000 CR3: 0000000038416003 CR4: 0000000000170ee0 [ 256.440678] Call Trace: [ 256.441573] ? show_regs.cold.9+0x22/0x2f [ 256.442931] ? __warn+0xc8/0x150 [ 256.444200] ? __list_add_valid+0xd7/0x100 [ 256.445328] ? report_bug+0x113/0x140 [ 256.446160] ? do_error_trap+0xb6/0x130 [ 256.446830] ? do_invalid_op+0x46/0x60 [ 256.447463] ? __list_add_valid+0xd7/0x100 [ 256.448470] ? invalid_op+0x14/0x20 [ 256.449412] ? __list_add_valid+0xd7/0x100 [ 256.450201] ldlm_namespace_move_to_active_locked+0x64/0xd0 [ptlrpc] [ 256.451886] ldlm_pools_count+0x56/0x1d0 [ptlrpc] [ 256.453189] ldlm_pools_srv_count+0x19/0x30 [ptlrpc] [ 256.454529] do_shrink_slab+0x41/0x530 [ 256.455139] shrink_slab+0xff/0x4f0 [ 256.456135] ? move_freepages_block+0x1f2/0x2b0 [ 256.457326] shrink_node+0x321/0xc20 [ 256.458469] do_try_to_free_pages+0xfc/0x7a0 [ 256.459685] try_to_free_pages+0x144/0x330 [ 256.460779] __alloc_pages_slowpath+0x59b/0x1870 [ 256.461895] ? blk_finish_plug+0x31/0x46 [ 256.462907] ? get_page_from_freelist+0x114/0x5d0 [ 256.464436] __alloc_pages_nodemask+0x4d3/0x550 [ 256.465485] abd_alloc_chunks+0x1b0/0x6b0 [zfs] [ 256.467021] ? __mutex_init+0x67/0x80 [ 256.467904] abd_alloc+0x5d/0x100 [zfs] [ 256.469249] arc_get_data_abd.isra.37+0x89/0xa0 [zfs] [ 256.470671] arc_hdr_alloc_abd+0x72/0x130 [zfs] [ 256.472128] arc_read+0xbd2/0x2650 [zfs] [ 256.473610] ? dbuf_issue_final_prefetch+0x190/0x190 [zfs] [ 256.475052] ? zfetch_kstats_update+0x110/0x110 [zfs] [ 256.477057] dbuf_issue_final_prefetch+0xc7/0x190 [zfs] [ 256.479338] dbuf_prefetch_impl+0x8c1/0x9e0 [zfs] [ 256.480603] ? zfetch_kstats_update+0x110/0x110 [zfs] [ 256.482532] dmu_zfetch_run+0x1c0/0x3a0 [zfs] [ 256.484434] dmu_buf_hold_array_by_dnode+0x33b/0x770 [zfs] [ 256.486099] dmu_buf_hold_array_by_bonus+0x1c/0x30 [zfs] [ 256.487727] osd_bufs_get_read+0x1e4/0x950 [osd_zfs] [ 256.488990] osd_bufs_get+0x141/0x1e0 [osd_zfs] [ 256.489872] ofd_preprw_read.isra.15+0x2f1/0x1770 [ofd] [ 256.491069] ofd_preprw+0x5e3/0xcb0 [ofd] [ 256.492210] tgt_brw_read+0x769/0x26c0 [ptlrpc] [ 256.493882] ? sptlrpc_svc_alloc_rs+0x70/0x460 [ptlrpc] [ 256.495331] ? lustre_msg_add_version+0x29/0xd0 [ptlrpc] [ 256.497154] ? lustre_pack_reply_flags+0x72/0x250 [ptlrpc] [ 256.498302] ? lustre_pack_reply+0x15/0x20 [ptlrpc] [ 256.499503] ? req_capsule_server_pack+0x250/0x3d0 [ptlrpc] [ 256.500798] tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 256.501893] tgt_request_handle+0x351/0x1c00 [ptlrpc] [ 256.503134] ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 256.504730] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 256.506088] ptlrpc_main+0xd1e/0x1440 [ptlrpc] [ 256.507294] ? ptlrpc_wait_event+0x980/0x980 [ptlrpc] [ 256.508388] kthread+0x1d1/0x200 [ 256.509438] ? set_kthread_struct+0x70/0x70 [ 256.510274] ret_from_fork+0x1f/0x30 [ 256.511085] ---[ end trace 2e86c68e9c14ac39 ]--- [ 500.760621] Lustre: DEBUG MARKER: SKIP: sanity-benchmark test_iozone skipping SLOW test iozone [ 501.530312] Lustre: DEBUG MARKER: == sanity-benchmark test fsx: fsx ======================== 13:50:07 (1752688207) [ 1075.211929] Lustre: DEBUG MARKER: == sanity-benchmark test fsx_partial_punch: Verify fsx with partial punch via fallocate ========================================================== 13:59:40 (1752688780) [ 1531.857673] Lustre: DEBUG MARKER: == sanity-benchmark test complete, duration 1409 sec ===== 14:07:17 (1752689237) [ 1532.363134] Lustre: DEBUG MARKER: === sanity-benchmark: start cleanup 14:07:18 (1752689238) === [ 1533.396466] Lustre: DEBUG MARKER: === sanity-benchmark: finish cleanup 14:07:19 (1752689239) === [ 1539.039546] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.039838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.044176] Lustre: Skipped 1 previous similar message [ 1540.482463] LustreError: 13382:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 1540.569532] Lustre: server umount lustre-MDT0000 complete [ 1541.956561] LustreError: 8141:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752689247 with bad export cookie 15973991188011419343 [ 1541.957564] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1541.961580] LustreError: 8141:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1541.981080] ------------[ cut here ]------------ [ 1541.982254] list_del corruption. prev->next should be ffff88b83d26fc58, but was ffffffffc18894f0 [ 1541.984180] WARNING: CPU: 1 PID: 13591 at lib/list_debug.c:53 __list_del_entry_valid+0xe9/0x140 [ 1541.985899] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1541.995674] CPU: 1 PID: 13591 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 1541.998704] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1542.001882] RIP: 0010:__list_del_entry_valid+0xe9/0x140 [ 1542.002886] Code: 83 05 02 7d 0c 03 01 c3 48 89 f2 48 89 fe 48 c7 c7 f0 e1 9e af 48 83 05 f4 7c 0c 03 01 e8 fb 48 8f ff 48 83 05 ef 7c 0c 03 01 <0f> 0b 48 83 05 ed 7c 0c 03 01 48 83 05 ed 7c 0c 03 01 31 c0 48 83 [ 1542.007264] RSP: 0018:ffffa1be41d4fae8 EFLAGS: 00010206 [ 1542.008490] RAX: 0000000000000000 RBX: ffff88b83d26fc00 RCX: 0000000000000000 [ 1542.010094] RDX: ffff88b8420ae640 RSI: ffff88b84209e5a8 RDI: ffff88b84209e5a8 [ 1542.011777] RBP: 0000000000000001 R08: 0000000000000000 R09: c0000000ffff7fff [ 1542.013351] R10: 0000000000000001 R11: ffffa1be41d4f8d8 R12: ffffa1be41d4fdf0 [ 1542.015039] R13: ffff88b840d55fc0 R14: 00000000fffffff4 R15: ffff88b8071bbe79 [ 1542.016822] FS: 00007f0fcbf29080(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 1542.018696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1542.019769] CR2: 00007ff569942860 CR3: 000000011a0c6003 CR4: 0000000000170ee0 [ 1542.021054] Call Trace: [ 1542.021623] ? show_regs.cold.9+0x22/0x2f [ 1542.022428] ? __warn+0xc8/0x150 [ 1542.023085] ? __list_del_entry_valid+0xe9/0x140 [ 1542.023972] ? report_bug+0x113/0x140 [ 1542.024726] ? do_error_trap+0xb6/0x130 [ 1542.025465] ? do_invalid_op+0x46/0x60 [ 1542.026108] ? __list_del_entry_valid+0xe9/0x140 [ 1542.026862] ? invalid_op+0x14/0x20 [ 1542.027768] ? __list_del_entry_valid+0xe9/0x140 [ 1542.028651] ? __list_del_entry_valid+0xe1/0x140 [ 1542.029615] ldlm_namespace_unregister+0x2c/0x90 [ptlrpc] [ 1542.030743] ldlm_namespace_free_post+0x43/0x320 [ptlrpc] [ 1542.032076] ofd_device_fini+0x1da/0x4a0 [ofd] [ 1542.032852] obd_precleanup+0x106/0x360 [obdclass] [ 1542.034258] ? class_disconnect_exports+0x36d/0x3a0 [obdclass] [ 1542.035548] class_cleanup+0x3fa/0xa80 [obdclass] [ 1542.036860] ? do_raw_spin_unlock+0x75/0x190 [ 1542.037554] class_process_config+0xd4f/0x2b60 [obdclass] [ 1542.038937] ? class_manual_cleanup+0x239/0xa20 [obdclass] [ 1542.040256] ? __kmalloc+0x1b4/0x4a0 [ 1542.041053] class_manual_cleanup+0x595/0xa20 [obdclass] [ 1542.042329] server_put_super+0x1049/0x1a90 [ptlrpc] [ 1542.043670] ? fsnotify_sb_delete+0x229/0x310 [ 1542.044713] generic_shutdown_super+0xb1/0x1b0 [ 1542.045584] kill_anon_super+0x1c/0x40 [ 1542.046510] lustre_kill_super+0x2a/0x60 [lustre] [ 1542.047691] deactivate_locked_super+0x52/0xd0 [ 1542.048604] deactivate_super+0x83/0x90 [ 1542.049501] cleanup_mnt+0x5f/0xe0 [ 1542.050183] __cleanup_mnt+0x16/0x20 [ 1542.051072] task_work_run+0xc6/0x110 [ 1542.051728] exit_to_usermode_loop+0x1dd/0x1f0 [ 1542.052613] do_syscall_64+0x3d6/0x440 [ 1542.053509] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1542.054391] RIP: 0033:0x7f0fcae7f8fb [ 1542.055138] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 1542.058687] RSP: 002b:00007fff745738e8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6 [ 1542.060367] RAX: 0000000000000000 RBX: 0000562b678baa40 RCX: 00007f0fcae7f8fb [ 1542.062208] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 0000562b678ce790 [ 1542.063644] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000562b678b9010 [ 1542.065508] R10: 0000000000000000 R11: 0000000000000202 R12: 0000562b678ce790 [ 1542.067262] R13: 00007f0fcbd03184 R14: 0000562b678bac20 R15: 00000000ffffffff [ 1542.068699] ---[ end trace 2e86c68e9c14ac3a ]--- [ 1542.076107] LustreError: 13591:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1542.078634] LustreError: 13591:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1542.085692] Lustre: server umount lustre-OST0000 complete [ 1542.303076] ------------[ cut here ]------------ [ 1542.304229] list_del corruption. prev->next should be ffff88b702d7a858, but was 5a5a5a5a5a5a5a5a [ 1542.306236] WARNING: CPU: 1 PID: 12445 at lib/list_debug.c:53 __list_del_entry_valid+0xe9/0x140 [ 1542.307758] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1542.320268] CPU: 1 PID: 12445 Comm: kworker/1:0 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 1542.322423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1542.324272] Workqueue: events ldlm_pools_recalc_task [ptlrpc] [ 1542.325989] RIP: 0010:__list_del_entry_valid+0xe9/0x140 [ 1542.327314] Code: 83 05 02 7d 0c 03 01 c3 48 89 f2 48 89 fe 48 c7 c7 f0 e1 9e af 48 83 05 f4 7c 0c 03 01 e8 fb 48 8f ff 48 83 05 ef 7c 0c 03 01 <0f> 0b 48 83 05 ed 7c 0c 03 01 48 83 05 ed 7c 0c 03 01 31 c0 48 83 [ 1542.331985] RSP: 0018:ffffa1be40c87df8 EFLAGS: 00010206 [ 1542.333038] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 1542.334943] RDX: ffff88b8420ae640 RSI: ffff88b84209e5a8 RDI: ffff88b84209e5a8 [ 1542.336913] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 1542.339499] R10: 0000000000000001 R11: ffffa1be40c87be8 R12: ffffffffc18894f0 [ 1542.341852] R13: 0000000000000001 R14: 0000000000000000 R15: ffffffffc181ed08 [ 1542.344091] FS: 0000000000000000(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 1542.346696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1542.348551] CR2: 00007ffd0194e0c0 CR3: 0000000038416001 CR4: 0000000000170ee0 [ 1542.350151] Call Trace: [ 1542.351093] ? show_regs.cold.9+0x22/0x2f [ 1542.351920] ? __warn+0xc8/0x150 [ 1542.352487] ? __list_del_entry_valid+0xe9/0x140 [ 1542.353634] ? report_bug+0x113/0x140 [ 1542.354606] ? do_error_trap+0xb6/0x130 [ 1542.355581] ? do_invalid_op+0x46/0x60 [ 1542.356382] ? __list_del_entry_valid+0xe9/0x140 [ 1542.357436] ? invalid_op+0x14/0x20 [ 1542.358516] ? __list_del_entry_valid+0xe9/0x140 [ 1542.359893] ? __list_del_entry_valid+0xe1/0x140 [ 1542.361199] ldlm_namespace_move_to_inactive_locked+0x36/0xd0 [ptlrpc] [ 1542.363094] ldlm_pools_recalc_delay+0x1b1/0x220 [ptlrpc] [ 1542.364612] ldlm_pools_recalc_task+0x1df/0x3c0 [ptlrpc] [ 1542.366284] process_one_work+0x2c8/0x700 [ 1542.367063] worker_thread+0x296/0x6e0 [ 1542.367880] ? rescuer_thread+0x570/0x570 [ 1542.368741] kthread+0x1d1/0x200 [ 1542.369587] ? set_kthread_struct+0x70/0x70 [ 1542.370312] ret_from_fork+0x1f/0x30 [ 1542.371110] ---[ end trace 2e86c68e9c14ac3b ]--- [ 1542.372372] ------------[ cut here ]------------ [ 1542.373540] list_add double add: new=ffff88b702d7a858, prev=ffff88b702d7a858, next=ffffffffc18894f0. [ 1542.375972] WARNING: CPU: 1 PID: 12445 at lib/list_debug.c:31 __list_add_valid+0xd7/0x100 [ 1542.378093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1542.389149] CPU: 1 PID: 12445 Comm: kworker/1:0 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 1542.392144] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1542.393706] Workqueue: events ldlm_pools_recalc_task [ptlrpc] [ 1542.395022] RIP: 0010:__list_add_valid+0xd7/0x100 [ 1542.396362] Code: 7b 0c 03 01 c3 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 10 e3 9e af 48 83 05 4e 7b 0c 03 01 e8 cd 47 8f ff 48 83 05 49 7b 0c 03 01 <0f> 0b 48 83 05 47 7b 0c 03 01 48 83 05 47 7b 0c 03 01 31 c0 48 83 [ 1542.400828] RSP: 0018:ffffa1be40c87df8 EFLAGS: 00010202 [ 1542.402139] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 1542.404141] RDX: ffff88b8420ae640 RSI: ffff88b84209e5a8 RDI: ffff88b84209e5a8 [ 1542.405863] RBP: ffff88b702d7a858 R08: 0000000000000000 R09: c0000000ffff7fff [ 1542.407800] R10: 0000000000000001 R11: ffffa1be40c87be8 R12: ffffffffc18894f0 [ 1542.409526] R13: ffff88b702d7a858 R14: 0000000000000000 R15: ffffffffc181ed08 [ 1542.411210] FS: 0000000000000000(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 1542.413263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1542.414749] CR2: 00007ffd0194e0c0 CR3: 0000000038416001 CR4: 0000000000170ee0 [ 1542.416548] Call Trace: [ 1542.416972] ? show_regs.cold.9+0x22/0x2f [ 1542.417633] ? __warn+0xc8/0x150 [ 1542.418392] ? __list_add_valid+0xd7/0x100 [ 1542.419285] ? report_bug+0x113/0x140 [ 1542.420106] ? do_error_trap+0xb6/0x130 [ 1542.420907] ? do_invalid_op+0x46/0x60 [ 1542.421604] ? __list_add_valid+0xd7/0x100 [ 1542.422444] ? invalid_op+0x14/0x20 [ 1542.423300] ? __list_add_valid+0xd7/0x100 [ 1542.424032] ? __list_add_valid+0xcf/0x100 [ 1542.424909] ldlm_namespace_move_to_inactive_locked+0x64/0xd0 [ptlrpc] [ 1542.426620] ldlm_pools_recalc_delay+0x1b1/0x220 [ptlrpc] [ 1542.428492] ldlm_pools_recalc_task+0x1df/0x3c0 [ptlrpc] [ 1542.430303] process_one_work+0x2c8/0x700 [ 1542.431527] worker_thread+0x296/0x6e0 [ 1542.432469] ? rescuer_thread+0x570/0x570 [ 1542.433374] kthread+0x1d1/0x200 [ 1542.434088] ? set_kthread_struct+0x70/0x70 [ 1542.434849] ret_from_fork+0x1f/0x30 [ 1542.435686] ---[ end trace 2e86c68e9c14ac3c ]--- [ 1543.395403] ------------[ cut here ]------------ [ 1543.396872] list_del corruption. prev->next should be ffff88b702d7a858, but was 5a5a5a5a5a5a5a5a [ 1543.399289] WARNING: CPU: 0 PID: 13793 at lib/list_debug.c:53 __list_del_entry_valid+0xe9/0x140 [ 1543.401501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1543.412189] CPU: 0 PID: 13793 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 1543.414220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1543.416299] RIP: 0010:__list_del_entry_valid+0xe9/0x140 [ 1543.417357] Code: 83 05 02 7d 0c 03 01 c3 48 89 f2 48 89 fe 48 c7 c7 f0 e1 9e af 48 83 05 f4 7c 0c 03 01 e8 fb 48 8f ff 48 83 05 ef 7c 0c 03 01 <0f> 0b 48 83 05 ed 7c 0c 03 01 48 83 05 ed 7c 0c 03 01 31 c0 48 83 [ 1543.421122] RSP: 0018:ffffa1be40ddbae8 EFLAGS: 00010202 [ 1543.422207] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 1543.424073] RDX: ffff88b84202e640 RSI: ffff88b84201e5a8 RDI: ffff88b84201e5a8 [ 1543.425523] RBP: 0000000000000001 R08: 0000000000000000 R09: c0000000ffff7fff [ 1543.427104] R10: 0000000000000001 R11: ffffa1be40ddb8d8 R12: ffffa1be40ddbdf0 [ 1543.428911] R13: ffff88b703ac0740 R14: 00000000fffffff4 R15: ffff88b81e38e9b9 [ 1543.430369] FS: 00007ffb00f6b080(0000) GS:ffff88b842000000(0000) knlGS:0000000000000000 [ 1543.431966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1543.433429] CR2: 000055e356724a38 CR3: 00000001039dc003 CR4: 0000000000170ef0 [ 1543.435057] Call Trace: [ 1543.435769] ? show_regs.cold.9+0x22/0x2f [ 1543.436839] ? __warn+0xc8/0x150 [ 1543.437406] ? __list_del_entry_valid+0xe9/0x140 [ 1543.438328] ? report_bug+0x113/0x140 [ 1543.438950] ? do_error_trap+0xb6/0x130 [ 1543.439786] ? do_invalid_op+0x46/0x60 [ 1543.440467] ? __list_del_entry_valid+0xe9/0x140 [ 1543.441420] ? invalid_op+0x14/0x20 [ 1543.442209] ? __list_del_entry_valid+0xe9/0x140 [ 1543.442996] ? __list_del_entry_valid+0xe1/0x140 [ 1543.444106] ldlm_namespace_unregister+0x2c/0x90 [ptlrpc] [ 1543.445987] ldlm_namespace_free_post+0x43/0x320 [ptlrpc] [ 1543.447379] ofd_device_fini+0x1da/0x4a0 [ofd] [ 1543.448368] obd_precleanup+0x106/0x360 [obdclass] [ 1543.449445] ? class_disconnect_exports+0x36d/0x3a0 [obdclass] [ 1543.450713] class_cleanup+0x3fa/0xa80 [obdclass] [ 1543.451929] ? do_raw_spin_unlock+0x75/0x190 [ 1543.452905] class_process_config+0xd4f/0x2b60 [obdclass] [ 1543.454195] ? class_manual_cleanup+0x239/0xa20 [obdclass] [ 1543.455532] ? __kmalloc+0x1b4/0x4a0 [ 1543.456114] ------------[ cut here ]------------ [ 1543.456195] class_manual_cleanup+0x595/0xa20 [obdclass] [ 1543.457607] list_del corruption. prev->next should be ffff88b702d7a858, but was 5a5a5a5a5a5a5a5a [ 1543.458762] server_put_super+0x1049/0x1a90 [ptlrpc] [ 1543.460616] WARNING: CPU: 1 PID: 12445 at lib/list_debug.c:53 __list_del_entry_valid+0xe9/0x140 [ 1543.461649] ? fsnotify_sb_delete+0x229/0x310 [ 1543.463435] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 1543.464381] generic_shutdown_super+0xb1/0x1b0 [ 1543.466191] mdd(O) mgs(O) osd_zfs(O) [ 1543.467267] kill_anon_super+0x1c/0x40 [ 1543.467272] lquota(O) [ 1543.467281] lustre_kill_super+0x2a/0x60 [lustre] [ 1543.468156] lfsck(O) [ 1543.469530] deactivate_locked_super+0x52/0xd0 [ 1543.470012] obdecho(O) [ 1543.471476] deactivate_super+0x83/0x90 [ 1543.472222] mgc(O) [ 1543.473099] cleanup_mnt+0x5f/0xe0 [ 1543.473591] mdc(O) [ 1543.474486] __cleanup_mnt+0x16/0x20 [ 1543.475066] lov(O) [ 1543.475766] task_work_run+0xc6/0x110 [ 1543.476498] osc(O) [ 1543.477286] exit_to_usermode_loop+0x1dd/0x1f0 [ 1543.477850] lmv(O) [ 1543.478666] do_syscall_64+0x3d6/0x440 [ 1543.479290] fid(O) [ 1543.480157] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1543.480876] fld(O) [ 1543.481925] RIP: 0033:0x7ffaffec18fb [ 1543.482310] ptlrpc_gss(O) [ 1543.483502] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 75 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 75 39 00 f7 d8 64 89 01 48 [ 1543.483945] ptlrpc(O) [ 1543.484833] RSP: 002b:00007ffecbf5b438 EFLAGS: 00000202 [ 1543.485634] obdclass(O) [ 1543.489504] ORIG_RAX: 00000000000000a6 [ 1543.489510] RAX: 0000000000000000 RBX: 0000559f697baa40 RCX: 00007ffaffec18fb [ 1543.490467] ksocklnd(O) [ 1543.491867] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 0000559f697ce410 [ 1543.492319] lnet(O) [ 1543.493418] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000559f697b9010 [ 1543.495544] libcfs(O) [ 1543.496118] R10: 0000000000000000 R11: 0000000000000202 R12: 0000559f697ce410 [ 1543.497610] zfs(O) [ 1543.498558] R13: 00007ffb00d45184 R14: 0000559f697bac20 R15: 00000000ffffffff [ 1543.499807] spl(O) [ 1543.500406] ---[ end trace 2e86c68e9c14ac3d ]--- [ 1543.501772] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1543.510258] LustreError: 13793:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1543.510797] CPU: 1 PID: 12445 Comm: kworker/1:0 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 1543.512812] LustreError: 13793:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1543.515248] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1543.515259] Workqueue: events ldlm_pools_recalc_task [ptlrpc] [ 1543.520125] LustreError: 13793:0:(ldlm_lockd.c:3541:ldlm_cleanup()) ldlm still has namespaces; clean these up first. [ 1543.521090] RIP: 0010:__list_del_entry_valid+0xe9/0x140 [ 1543.521106] Code: 83 05 02 7d 0c 03 01 c3 48 89 f2 48 89 fe 48 c7 c7 f0 e1 9e af 48 83 05 f4 7c 0c 03 01 e8 fb 48 8f ff 48 83 05 ef 7c 0c 03 01 <0f> 0b 48 83 05 ed 7c 0c 03 01 48 83 05 ed 7c 0c 03 01 31 c0 48 83 [ 1543.521110] RSP: 0018:ffffa1be40c87df8 EFLAGS: 00010202 [ 1543.524232] LustreError: 13793:0:(lustre_dlm.h:590:ns_is_client()) ASSERTION( ns->ns_client == LDLM_NAMESPACE_CLIENT || ns->ns_client == LDLM_NAMESPACE_SERVER ) failed: [ 1543.525109] [ 1543.525113] RAX: 0000000000000000 RBX: ffff88b702d7a800 RCX: 0000000000000000 [ 1543.525116] RDX: ffff88b8420ae640 RSI: ffff88b84209e5a8 RDI: ffff88b84209e5a8 [ 1543.525119] RBP: ffff88b702d7a858 R08: 2e6e6f6974707572 R09: 726f63206c65645f [ 1543.525122] R10: 74707572726f6320 R11: 6c65645f7473696c R12: ffffffffc18894f0 [ 1543.525124] R13: 0000000000000001 R14: 0000000000000000 R15: ffffffffc181ed08 [ 1543.525127] FS: 0000000000000000(0000) GS:ffff88b842080000(0000) knlGS:0000000000000000 [ 1543.525129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1543.525132] CR2: 00007f5e1fcd0000 CR3: 0000000038416006 CR4: 0000000000170ee0 [ 1543.525140] Call Trace: [ 1543.525151] ? show_regs.cold.9+0x22/0x2f [ 1543.530790] LustreError: 13793:0:(lustre_dlm.h:590:ns_is_client()) LBUG [ 1543.532777] ? __warn+0xc8/0x150 [ 1543.536355] CPU: 3 PID: 13793 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 1543.536974] ? __list_del_entry_valid+0xe9/0x140 [ 1543.538623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1543.540161] ? report_bug+0x113/0x140 [ 1543.542426] Call Trace: [ 1543.544172] ? do_error_trap+0xb6/0x130 [ 1543.545455] ? dump_stack+0xbb/0x10e [ 1543.547901] ? do_invalid_op+0x46/0x60 [ 1543.549396] ? lbug_with_loc.cold.4+0xd/0x86 [libcfs] [ 1543.551080] ? __list_del_entry_valid+0xe9/0x140 [ 1543.551963] ? ldlm_namespace_dump+0x211/0x220 [ptlrpc] [ 1543.552803] ? invalid_op+0x14/0x20 [ 1543.554780] ? ldlm_dump_all_namespaces+0x58/0xb0 [ptlrpc] [ 1543.555830] ? __list_del_entry_valid+0xe9/0x140 [ 1543.558408] ? ldlm_cleanup+0xcf/0x7a0 [ptlrpc] [ 1543.560110] ? __list_del_entry_valid+0xe1/0x140 [ 1543.561943] ? update_load_avg+0x9f/0xa40 [ 1543.562920] ldlm_namespace_move_to_inactive_locked+0x36/0xd0 [ptlrpc] [ 1543.563604] ? update_load_avg+0x9f/0xa40 [ 1543.564434] ldlm_pools_recalc_delay+0x1b1/0x220 [ptlrpc] [ 1543.565603] ? set_next_entity+0xaa/0xc0 [ 1543.566677] ldlm_pools_recalc_task+0x1df/0x3c0 [ptlrpc] [ 1543.568009] ? do_raw_spin_unlock+0x75/0x190 [ 1543.568899] process_one_work+0x2c8/0x700 [ 1543.570931] ? _raw_spin_unlock_irq+0x12/0x50 [ 1543.571605] worker_thread+0x296/0x6e0 [ 1543.573696] ? finish_task_switch+0x125/0x560 [ 1543.575103] ? rescuer_thread+0x570/0x570 [ 1543.576771] ? __schedule+0x369/0xcb0 [ 1543.577793] kthread+0x1d1/0x200 [ 1543.579422] ? ldlm_put_ref+0x156/0x1e0 [ptlrpc] [ 1543.580658] ? set_kthread_struct+0x70/0x70 [ 1543.582243] ? client_obd_cleanup+0x11f/0x4d0 [ptlrpc] [ 1543.583795] ret_from_fork+0x1f/0x30 [ 1543.585237] ? lwp_device_fini+0x90/0x1e0 [osp] [ 1543.586463] ---[ end trace 2e86c68e9c14ac3e ]--- [ 1543.587971] ? obd_precleanup+0x106/0x360 [obdclass] [ 1543.589465] ------------[ cut here ]------------ [ 1543.591114] ? class_disconnect_exports+0x1bd/0x3a0 [obdclass] [ 1543.592032] list_add corruption. prev->next should be next (ffffffffc18894f0), but was 5a5a5a5a5a5a5a5a. (prev=ffff88b702d7a858). [ 1543.593643] ? class_cleanup+0x3fa/0xa80 [obdclass] [ 1543.594931] WARNING: CPU: 1 PID: 12445 at lib/list_debug.c:28 __list_add_valid+0x95/0x100 [ 1543.596139] ? do_raw_spin_unlock+0x75/0x190 [ 1543.597189] Modules linked in: [ 1543.598603] ? class_process_config+0xd4f/0x2b60 [obdclass] [ 1543.599872] lustre(O) [ 1543.601678] ? class_manual_cleanup+0x239/0xa20 [obdclass] [ 1543.602492] osp(O) [ 1543.603645] ? __kmalloc+0x1b4/0x4a0 [ 1543.604648] ofd(O) [ 1543.606310] ? class_manual_cleanup+0x595/0xa20 [obdclass] [ 1543.608057] lod(O) [ 1543.609989] ? server_put_super+0xc80/0x1a90 [ptlrpc] [ 1543.612807] mdt(O) [ 1543.613755] ? fsnotify_sb_delete+0x229/0x310 [ 1543.615616] mdd(O) [ 1543.616583] ? generic_shutdown_super+0xb1/0x1b0 [ 1543.617383] mgs(O) [ 1543.618628] ? kill_anon_super+0x1c/0x40 [ 1543.619307] osd_zfs(O) [ 1543.620815] ? lustre_kill_super+0x2a/0x60 [lustre] [ 1543.621202] lquota(O) [ 1543.622250] ? deactivate_locked_super+0x52/0xd0 [ 1543.622755] lfsck(O) [ 1543.624477] ? deactivate_super+0x83/0x90 [ 1543.625095] obdecho(O) [ 1543.626582] ? cleanup_mnt+0x5f/0xe0 [ 1543.627075] mgc(O) [ 1543.628145] ? __cleanup_mnt+0x16/0x20 [ 1543.628577] mdc(O) [ 1543.630211] ? task_work_run+0xc6/0x110 [ 1543.630835] lov(O) [ 1543.631937] ? exit_to_usermode_loop+0x1dd/0x1f0 [ 1543.632779] osc(O) [ 1543.633957] ? do_syscall_64+0x3d6/0x440 [ 1543.634465] lmv(O) [ 1543.635507] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1543.636012] fid(O) [ 1543.637258] Kernel panic - not syncing: LBUG [ 1543.638024] fld(O) [ 1543.638649] CPU: 3 PID: 13793 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 1543.639527] ptlrpc_gss(O) [ 1543.640488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1543.641388] ptlrpc(O) [ 1543.642262] Call Trace: [ 1543.643069] obdclass(O) [ 1543.643967] ? dump_stack+0xbb/0x10e [ 1543.644682] ksocklnd(O) [ 1543.645612] ? panic+0x142/0x44c [ 1543.646384] lnet(O) [ 1543.647321] ? lbug_with_loc.cold.4+0x32/0x86 [libcfs] [ 1543.648079] libcfs(O) [ 1543.649115] ? ldlm_namespace_dump+0x211/0x220 [ptlrpc] [ 1543.649862] zfs(O) [ 1543.652127] ? ldlm_dump_all_namespaces+0x58/0xb0 [ptlrpc] [ 1543.652781] spl(O) [ 1543.655284] ? ldlm_cleanup+0xcf/0x7a0 [ptlrpc] [ 1543.655795] rpcsec_gss_krb5 [ 1543.656217] ? update_load_avg+0x9f/0xa40 [ 1543.656762] auth_rpcgss [ 1543.657353] ? update_load_avg+0x9f/0xa40 [ 1543.657888] nfsv4 [ 1543.658414] ? set_next_entity+0xaa/0xc0 [ 1543.658841] dns_resolver [ 1543.659862] ? do_raw_spin_unlock+0x75/0x190 [ 1543.660399] intel_rapl_msr [ 1543.661927] ? _raw_spin_unlock_irq+0x12/0x50 [ 1543.662393] intel_rapl_common [ 1543.663743] ? finish_task_switch+0x125/0x560 [ 1543.664224] sb_edac [ 1543.665001] ? __schedule+0x369/0xcb0 [ 1543.665570] rapl [ 1543.666313] ? ldlm_put_ref+0x156/0x1e0 [ptlrpc] [ 1543.666698] pcspkr [ 1543.668162] ? client_obd_cleanup+0x11f/0x4d0 [ptlrpc] [ 1543.668641] i2c_piix4 [ 1543.669547] ? lwp_device_fini+0x90/0x1e0 [osp] [ 1543.670245] squashfs [ 1543.671620] ? obd_precleanup+0x106/0x360 [obdclass] [ 1543.672573] crct10dif_pclmul [ 1543.673301] ? class_disconnect_exports+0x1bd/0x3a0 [obdclass] [ 1543.674435] crc32_pclmul [ 1543.675462] ? class_cleanup+0x3fa/0xa80 [obdclass] [ 1543.675943] crc32c_intel [ 1543.676625] ? do_raw_spin_unlock+0x75/0x190 [ 1543.676989] ata_generic [ 1543.678184] ? class_process_config+0xd4f/0x2b60 [obdclass] [ 1543.678566] ghash_clmulni_intel [ 1543.679767] ? class_manual_cleanup+0x239/0xa20 [obdclass] [ 1543.680395] serio_raw [ 1543.681470] ? __kmalloc+0x1b4/0x4a0 [ 1543.682053] ata_piix [ 1543.683219] ? class_manual_cleanup+0x595/0xa20 [obdclass] [ 1543.683784] libata [ 1543.685127] ? server_put_super+0xc80/0x1a90 [ptlrpc] [ 1543.685824] dm_mirror [ 1543.687253] ? fsnotify_sb_delete+0x229/0x310 [ 1543.688241] dm_region_hash [ 1543.689245] ? generic_shutdown_super+0xb1/0x1b0 [ 1543.689864] dm_log [ 1543.691247] ? kill_anon_super+0x1c/0x40 [ 1543.691761] dm_mod [ 1543.692683] ? lustre_kill_super+0x2a/0x60 [lustre] [ 1543.693131] sha512_ssse3 [ 1543.693742] ? deactivate_locked_super+0x52/0xd0 [ 1543.694114] sha512_generic [ 1543.695134] ? deactivate_super+0x83/0x90 [ 1543.695492] [ 1543.696385] ? cleanup_mnt+0x5f[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-957.1.3.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Thu Nov 29 14:49:43 UTC 2018 [ 0.000000] Command line: rd.shell ro panic=1 nomodeset ipmtu=9000 rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr disable_cpu_apicid=0 acpi_rsdp=0xf5950 elfcorehdr=3079540K [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ac000000-0x00000000bbf5cfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bbfffc00-0x00000000bbffffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] extended physical RAM map: [ 0.000000] reserve setup_data: [mem 0x0000000000000000-0x0000000000000fff] reserved [ 0.000000] reserve setup_data: [mem 0x0000000000001000-0x000000000009fbff] usable [ 0.000000] reserve setup_data: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000ac000000-0x00000000ac00002f] usable [ 0.000000] reserve setup_data: [mem 0x00000000ac000030-0x00000000bbf5cfff] usable [ 0.000000] reserve setup_data: [mem 0x00000000bbfffc00-0x00000000bbffffff] usable [ 0.000000] reserve setup_data: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0xbc000 max_arch_pfn = 0x400000000 [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] x2apic enabled by BIOS, switching to x2apic ops [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0xb77a1000-0xba7fffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] Setting APIC routing to cluster x2apic. [ 0.000000] NUMA turned off [ 0.000000] Faking a node at [mem 0x0000000000000000-0x00000000bbffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0xbbf36000-0xbbf5cfff] [ 0.000000] kvm-clock: cpu 0, msr 0:bbee5001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1545779292130 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0xac000000-0xbbf5cfff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0xbbf5cfff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: Disabling requested cpu. Processor 0/0x0 ignored. [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: NR_CPUS/possible_cpus limit of 1 almost reached. Keeping one slot for boot cpu. Processor 1/0x1 ignored. [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: NR_CPUS/possible_cpus limit of 1 almost reached. Keeping one slot for boot cpu. Processor 2/0x2 ignored. [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: 4 Processors exceeds NR_CPUS limit of 1 [ 0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x00100000-0xabffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xac000000-0xac000fff] [ 0.000000] PM: Registered nosave memory: [mem 0xbbf5d000-0xbbffffff] [ 0.000000] e820: [mem 0x00100000-0xabffffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 38 pages/cpu @ffff8800bbc00000 s118784 r8192 d28672 u2097152 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr bbc1c040 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 64485 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: rd.shell ro panic=1 nomodeset ipmtu=9000 rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr disable_cpu_apicid=0 acpi_rsdp=0xf5950 elfcorehdr=3079540K [ 0.000000] audit: disabled (until reboot) [ 0.000000] Misrouted IRQ fixup and polling support enabled [ 0.000000] This may significantly impact system performance [ 0.000000] Disabling memory control group subsystem [ 0.000000] PID hash table entries: 1024 (order: 1, 8192 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 184636k/3080192k available (7664k kernel code, 2818068k absent, 77488k reserved, 6055k data, 1876k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] x86/pti: Unmapping kernel while in userspace [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1. [ 0.000000] NR_IRQS:327936 nr_irqs:256 0 [ 0.000000] do_IRQ: 0.236 No irq handler for vector (irq -1) [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.211138] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.214236] pid_max: default: 32768 minimum: 301 [ 0.215936] Security Framework initialized [ 0.217296] SELinux: Initializing. [ 0.218259] Yama: becoming mindful. [ 0.219785] Dentry cache hash table entries: 32768 (order: 6, 262144 bytes) [ 0.221993] Inode-cache hash table entries: 16384 (order: 5, 131072 bytes) [ 0.223964] Mount-cache hash table entries: 512 (order: 0, 4096 bytes) [ 0.226415] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes) [ 0.228914] Initializing cgroup subsys memory [ 0.230059] Initializing cgroup subsys devices [ 0.231542] Initializing cgroup subsys freezer [ 0.232933] Initializing cgroup subsys net_cls [ 0.234145] Initializing cgroup subsys blkio [ 0.235169] Initializing cgroup subsys perf_event [ 0.236235] Initializing cgroup subsys hugetlb [ 0.237514] Initializing cgroup subsys pids [ 0.238962] Initializing cgroup subsys net_prio [ 0.240744] x86/cpu: Activated the Intel User Mode Instruction Prevention (UMIP) CPU feature [ 0.244960] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.246527] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.248439] tlb_flushall_shift: 6 [ 0.249726] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.253041] FEATURE SPEC_CTRL Present [ 0.254325] FEATURE IBPB_SUPPORT Present [ 0.256119] Spectre V2 : Mitigation: Full retpoline [ 0.284420] Freeing SMP alternatives: 28k freed [ 0.296264] ACPI: Core revision 20130517 [ 0.298335] ACPI: All ACPI Tables successfully acquired [ 0.300041] ftrace: allocating 29188 entries in 115 pages [ 0.359899] Switched APIC routing to physical x2apic. [ 0.362657] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.363865] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.366187] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.368166] ... version: 2 [ 0.369301] ... bit width: 48 [ 0.370292] ... generic registers: 4 [ 0.371005] ... value mask: 0000ffffffffffff [ 0.372137] ... max period: 00007fffffffffff [ 0.373295] ... fixed-purpose events: 3 [ 0.374165] ... event mask: 000000070000000f [ 0.375288] KVM setup paravirtual spinlock [ 0.377566] Brought up 1 CPUs [ 0.378236] smpboot: Max logical packages: 1 [ 0.379186] smpboot: Total of 1 processors activated (4799.99 BogoMIPS) [ 0.381666] devtmpfs: initialized [ 0.385085] EVM: security.selinux [ 0.385933] EVM: security.ima [ 0.386516] EVM: security.capability [ 0.388936] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.390976] pinctrl core: initialized pinctrl subsystem [ 0.392127] RTC time: 18:07:30, date: 07/16/25 [ 0.392937] NET: Registered protocol family 16 [ 0.394215] ACPI: bus type PCI registered [ 0.395065] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.396451] PCI: Using configuration type 1 for base access [ 0.397579] core: PMU erratum BJ122, BV98, HSD29 workaround disabled, HT off [ 0.400381] ACPI: Added _OSI(Module Device) [ 0.401224] ACPI: Added _OSI(Processor Device) [ 0.401992] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.403412] ACPI: Added _OSI(Processor Aggregator Device) [ 0.404548] ACPI: Added _OSI(Linux-Dell-Video) [ 0.406671] ACPI: Interpreter enabled [ 0.407691] ACPI: (supports S0 S3 S4 S5) [ 0.408580] ACPI: Using IOAPIC for interrupt routing [ 0.409711] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.411764] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.416112] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.417552] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.419042] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.420683] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.423217] acpiphp: Slot [2] registered [ 0.424068] acpiphp: Slot [3] registered [ 0.424869] acpiphp: Slot [4] registered [ 0.425638] acpiphp: Slot [5] registered [ 0.426415] acpiphp: Slot [6] registered [ 0.427184] acpiphp: Slot [7] registered [ 0.427939] acpiphp: Slot [8] registered [ 0.428905] acpiphp: Slot [9] registered [ 0.429854] acpiphp: Slot [10] registered [ 0.430560] acpiphp: Slot [11] registered [ 0.431683] acpiphp: Slot [12] registered [ 0.432418] acpiphp: Slot [13] registered [ 0.433362] acpiphp: Slot [14] registered [ 0.434159] acpiphp: Slot [15] registered [ 0.435265] acpiphp: Slot [16] registered [ 0.436472] acpiphp: Slot [17] registered [ 0.437707] acpiphp: Slot [18] registered [ 0.438735] acpiphp: Slot [19] registered [ 0.439778] acpiphp: Slot [20] registered [ 0.440772] acpiphp: Slot [21] registered [ 0.441828] acpiphp: Slot [22] registered [ 0.442710] acpiphp: Slot [23] registered [ 0.443810] acpiphp: Slot [24] registered [ 0.444849] acpiphp: Slot [25] registered [ 0.445901] acpiphp: Slot [26] registered [ 0.447083] acpiphp: Slot [27] registered [ 0.447879] acpiphp: Slot [28] registered [ 0.449002] acpiphp: Slot [29] registered [ 0.450202] acpiphp: Slot [30] registered [ 0.451130] acpiphp: Slot [31] registered [ 0.452318] PCI host bridge to bus 0000:00 [ 0.453630] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.454824] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.456963] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.459573] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.462256] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.465138] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.475719] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.477846] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.479516] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.482245] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.484761] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.486740] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.667603] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 10 *11) [ 0.669104] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.670469] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.671664] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 *10 11) [ 0.672925] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.674633] ACPI: NR_CPUS/possible_cpus limit of 1 reached. Processor 4/0x1 ignored. [ 0.677793] ACPI: Unable to map lapic to logical cpu number [ 0.680041] ACPI: NR_CPUS/possible_cpus limit of 1 reached. Processor 5/0x2 ignored. [ 0.682921] ACPI: Unable to map lapic to logical cpu number [ 0.685148] acpi LNXCPU:03: BIOS reported wrong ACPI id 0 for the processor [ 0.687867] vgaarb: loaded [ 0.688977] SCSI subsystem initialized [ 0.690293] ACPI: bus type USB registered [ 0.691223] usbcore: registered new interface driver usbfs [ 0.692228] usbcore: registered new interface driver hub [ 0.693577] usbcore: registered new device driver usb [ 0.695686] EDAC MC: Ver: 3.0.0 [ 0.696886] PCI: Using ACPI for IRQ routing [ 0.698490] NetLabel: Initializing [ 0.699380] NetLabel: domain hash size = 128 [ 0.700482] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.702260] NetLabel: unlabeled traffic allowed by default [ 0.703523] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.704718] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.711078] amd_nb: Cannot enumerate AMD northbridges [ 0.713121] Switched to clocksource kvm-clock [ 0.722061] pnp: PnP ACPI init [ 0.722800] ACPI: bus type PNP registered [ 0.724161] pnp: PnP ACPI: found 6 devices [ 0.724994] ACPI: bus type PNP unregistered [ 0.732890] NET: Registered protocol family 2 [ 0.735024] TCP established hash table entries: 2048 (order: 2, 16384 bytes) [ 0.737044] TCP bind hash table entries: 2048 (order: 3, 32768 bytes) [ 0.738433] TCP: Hash tables configured (established 2048 bind 2048) [ 0.739749] TCP: reno registered [ 0.740590] UDP hash table entries: 256 (order: 1, 8192 bytes) [ 0.742016] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes) [ 0.743583] NET: Registered protocol family 1 [ 0.745171] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.747507] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.749735] Unpacking initramfs... [ 0.798174] Freeing initrd memory: 49532k freed [ 0.820461] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 0.822411] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 0.823971] RAPL PMU: hw unit of domain package 2^-0 Joules [ 0.825974] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 0.827378] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.828688] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.831657] futex hash table entries: 256 (order: 2, 16384 bytes) [ 0.833415] Initialise system trusted keyring [ 0.861721] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.863694] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.868336] zpool: loaded [ 0.868883] zbud: loaded [ 0.869704] VFS: Disk quotas dquot_6.5.2 [ 0.870675] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.872458] msgmni has been set to 457 [ 0.873340] Key type big_key registered [ 0.875511] NET: Registered protocol family 38 [ 0.876638] Key type asymmetric registered [ 0.877510] Asymmetric key parser 'x509' registered [ 0.879368] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 0.881961] io scheduler noop registered [ 0.882723] io scheduler deadline registered (default) [ 0.884144] io scheduler cfq registered [ 0.885209] io scheduler mq-deadline registered [ 0.886255] io scheduler kyber registered [ 0.887464] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.888624] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.890272] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 0.891937] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.893680] ACPI: Power Button [PWRF] [ 0.894838] GHES: HEST is not enabled! [ 0.895894] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.923910] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 0.952981] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.955145] Non-volatile memory driver v1.3 [ 0.956038] Linux agpgart interface v0.103 [ 0.956868] crash memory driver: version 1.1 [ 0.957686] rdac: device handler registered [ 0.958937] hp_sw: device handler registered [ 0.960123] emc: device handler registered [ 0.961072] alua: device handler registered [ 0.961849] libphy: Fixed MDIO Bus: probed [ 0.962779] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.963938] ehci-pci: EHCI PCI platform driver [ 0.965414] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.967239] ohci-pci: OHCI PCI platform driver [ 0.968160] uhci_hcd: USB Universal Host Controller Interface driver [ 0.969921] usbcore: registered new interface driver usbserial_generic [ 0.972318] usbserial: USB Serial support registered for generic [ 0.974238] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.977019] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.978409] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.980330] mousedev: PS/2 mouse device common for all mice [ 0.982725] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.986041] rtc_cmos 00:05: RTC can wake from S4 [ 0.988285] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.991396] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 0.993667] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 0.996615] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.999383] cpuidle: using governor menu [ 1.001355] hidraw: raw HID events driver (C) Jiri Kosina [ 1.003276] usbcore: registered new interface driver usbhid [ 1.004960] usbhid: USB HID core driver [ 1.006345] drop_monitor: Initializing network drop monitor service [ 1.008253] TCP: cubic registered [ 1.009403] Initializing XFRM netlink socket [ 1.010832] NET: Registered protocol family 10 [ 1.012406] NET: Registered protocol family 17 [ 1.013835] mpls_gso: MPLS GSO support [ 1.014969] mce: Unable to init device /dev/mcelog (rc: -5) [ 1.017090] Loading compiled-in X.509 certificates [ 1.018613] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 1.021468] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 1.025263] Loaded X.509 cert 'CentOS Linux kernel signing key: e7cef3613a9b8bd012fae7498272159bb1879c65' [ 1.027860] registered taskstats version 1 [ 1.033003] Key type trusted registered [ 1.037578] Key type encrypted registered [ 1.039011] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 1.043416] Magic number: 1:184:140 [ 1.044629] acpi device:11: hash matches [ 1.045721] rtc_cmos 00:05: setting system clock to 2025-07-16 18:07:30 UTC (1752689250) [ 1.049514] Freeing unused kernel memory: 1876k freed [ 1.051430] Write protecting the kernel read-only data: 12288k [ 1.054136] Freeing unused kernel memory: 516k freed [ 1.057021] Freeing unused kernel memory: 600k freed [ 1.081653] loop: module loaded [ 1.090289] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 1.110242] SQUASHFS error: Filesystem uses "unknown" compression. This is not supported mount: /squash/root: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error. [ 1.121345] TECH PREVIEW: Overlay filesystem may not be fully supported. [ 1.121345] Please review provided documentation for limitations. [ 1.126189] overlayfs: upper fs does not support xattr. mount: /newroot/squash: mount point does not exist. switch_root: cannot access /init: No such file or directory switch_root: failed to execute /init: No such file or directory [ 1.140434] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 [ 1.140434] [ 1.143806] CPU: 0 PID: 1 Comm: switch_root Tainted: G ------------ T 3.10.0-957.1.3.el7.x86_64 #1 [ 1.147539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1.150803] Call Trace: [ 1.151690] [] dump_stack+0x19/0x1b [ 1.153591] [] panic+0xe8/0x21f [ 1.155401] [] do_exit+0xa36/0xa40 [ 1.157254] [] do_group_exit+0x3f/0xa0 [ 1.159172] [] SyS_exit_group+0x14/0x20 [ 1.161226] [] system_call_fastpath+0x22/0x27 [ 1.163508] Kernel Offset: disabled [ 1.164852] Rebooting in 10 seconds..