[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 753630622 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002229] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009144] LSM: Security Framework initializing [ 0.010047] Yama: becoming mindful. [ 0.011038] SELinux: Initializing. [ 0.012073] *** VALIDATE selinux *** [ 0.019806] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024594] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026092] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027109] *** VALIDATE tmpfs *** [ 0.029267] *** VALIDATE proc *** [ 0.030204] *** VALIDATE cgroup *** [ 0.031007] *** VALIDATE cgroup2 *** [ 0.032260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036020] Spectre V2 : User space: Vulnerable [ 0.037005] Speculative Store Bypass: Vulnerable [ 0.040158] debug: unmapping init [mem 0xffffffffbc059000-0xffffffffbc060fff] [ 0.042142] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043525] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044018] ... version: 2 [ 0.045010] ... bit width: 48 [ 0.046009] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048011] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050013] ... event mask: 000000070000000f [ 0.051298] rcu: Hierarchical SRCU implementation. [ 0.053445] smp: Bringing up secondary CPUs ... [ 0.054551] x86: Booting SMP configuration: [ 0.055020] .... node #0, CPUs: #1 #2 #3 [ 0.058114] smp: Brought up 1 node, 4 CPUs [ 0.060015] smpboot: Max logical packages: 1 [ 0.061013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.182295] node 0 deferred pages initialised in 118ms [ 0.185211] devtmpfs: initialized [ 0.186182] x86/mm: Memory block size: 128MB [ 0.188738] gcov: version magic: 0x41383552 [ 0.189622] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.193127] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.195439] pinctrl core: initialized pinctrl subsystem [ 0.197356] [ 0.197852] ************************************************************* [ 0.201014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203013] ** ** [ 0.205026] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.207018] ** ** [ 0.210015] ** This means that this kernel is built to expose internal ** [ 0.212015] ** IOMMU data structures, which may compromise security on ** [ 0.214015] ** your system. ** [ 0.216014] ** ** [ 0.219016] ** If you see this message and you are not debugging the ** [ 0.221013] ** kernel, report this immediately to your vendor! ** [ 0.223025] ** ** [ 0.225015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.227014] ************************************************************* [ 0.230582] NET: Registered protocol family 16 [ 0.232492] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.235059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.238056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.242107] cpuidle: using governor menu [ 0.243674] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.245452] PCI: Using configuration type 1 for base access [ 0.246088] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.254106] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.255031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.257058] cryptd: max_cpu_qlen set to 1000 [ 0.258288] ACPI: Added _OSI(Module Device) [ 0.260010] ACPI: Added _OSI(Processor Device) [ 0.261010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.262007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.266000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.272531] ACPI: Interpreter enabled [ 0.274056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.276024] ACPI: Using IOAPIC for interrupt routing [ 0.277130] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.281152] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.291064] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.293044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.296023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.299109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.304475] acpiphp: Slot [2] registered [ 0.306250] acpiphp: Slot [5] registered [ 0.307113] acpiphp: Slot [6] registered [ 0.309218] acpiphp: Slot [7] registered [ 0.310126] acpiphp: Slot [8] registered [ 0.311121] acpiphp: Slot [9] registered [ 0.313120] acpiphp: Slot [10] registered [ 0.314126] acpiphp: Slot [3] registered [ 0.315082] acpiphp: Slot [4] registered [ 0.317105] acpiphp: Slot [11] registered [ 0.318184] acpiphp: Slot [12] registered [ 0.320133] acpiphp: Slot [13] registered [ 0.321098] acpiphp: Slot [14] registered [ 0.323104] acpiphp: Slot [15] registered [ 0.324088] acpiphp: Slot [16] registered [ 0.325086] acpiphp: Slot [17] registered [ 0.326080] acpiphp: Slot [18] registered [ 0.328111] acpiphp: Slot [19] registered [ 0.330151] acpiphp: Slot [20] registered [ 0.332098] acpiphp: Slot [21] registered [ 0.334088] acpiphp: Slot [22] registered [ 0.335012] acpiphp: Slot [23] registered [ 0.336149] acpiphp: Slot [24] registered [ 0.337093] acpiphp: Slot [25] registered [ 0.339119] acpiphp: Slot [26] registered [ 0.340149] acpiphp: Slot [27] registered [ 0.342084] acpiphp: Slot [28] registered [ 0.343077] acpiphp: Slot [29] registered [ 0.343944] acpiphp: Slot [30] registered [ 0.344061] acpiphp: Slot [31] registered [ 0.344902] PCI host bridge to bus 0000:00 [ 0.346029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.348028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.350023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.352035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.354026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.356029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.360975] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.363211] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.371018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.375482] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377037] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.381021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.383456] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.385574] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389586] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.404015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.406010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409135] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.420013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.441017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.452646] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.486044] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.590036] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.629934] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.647020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.664024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.696028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.715806] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.734025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.760027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.835034] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.872649] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.891020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.907030] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.962038] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.984144] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.019034] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.055023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.141023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.160560] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.163329] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.166378] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.169547] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.171271] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.177080] iommu: Default domain type: Passthrough [ 1.179481] SCSI subsystem initialized [ 1.181155] ACPI: bus type USB registered [ 1.183142] usbcore: registered new interface driver usbfs [ 1.185066] usbcore: registered new interface driver hub [ 1.187096] usbcore: registered new device driver usb [ 1.190214] pps_core: LinuxPPS API ver. 1 registered [ 1.191008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.194061] PTP clock support registered [ 1.196257] EDAC MC: Ver: 3.0.0 [ 1.198875] PCI: Using ACPI for IRQ routing [ 1.201916] NetLabel: Initializing [ 1.203009] NetLabel: domain hash size = 128 [ 1.204011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.207125] NetLabel: unlabeled traffic allowed by default [ 1.209143] vgaarb: loaded [ 1.210281] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.212014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.219127] clocksource: Switched to clocksource kvm-clock [ 1.332282] VFS: Disk quotas dquot_6.6.0 [ 1.333703] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.336092] *** VALIDATE ramfs *** [ 1.337328] *** VALIDATE hugetlbfs *** [ 1.338818] pnp: PnP ACPI init [ 1.341227] pnp: PnP ACPI: found 6 devices [ 1.358304] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.361903] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.364043] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.366444] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.368853] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.371455] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.374291] NET: Registered protocol family 2 [ 1.376842] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.381529] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.385200] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.390869] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.394570] TCP: Hash tables configured (established 65536 bind 65536) [ 1.397745] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.401133] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.404194] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.407414] NET: Registered protocol family 1 [ 1.409961] RPC: Registered named UNIX socket transport module. [ 1.412871] RPC: Registered udp transport module. [ 1.415232] RPC: Registered tcp transport module. [ 1.417891] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.420767] NET: Registered protocol family 44 [ 1.423163] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.426063] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.429185] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.432324] PCI: CLS 0 bytes, default 64 [ 1.434245] Unpacking initramfs... [ 2.803173] debug: unmapping init [mem 0xffff8a033cc54000-0xffff8a033ffbffff] [ 2.807100] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.809407] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.812355] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.283745] Initialise system trusted keyrings [ 3.285279] Key type blacklist registered [ 3.286851] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.295205] zbud: loaded [ 3.298276] *** VALIDATE nfs *** [ 3.299034] *** VALIDATE nfs4 *** [ 3.300949] pstore: using deflate compression [ 3.304031] Platform Keyring initialized [ 3.422644] NET: Registered protocol family 38 [ 3.424406] Key type asymmetric registered [ 3.425589] Asymmetric key parser 'x509' registered [ 3.427417] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.429925] io scheduler mq-deadline registered [ 3.430627] io scheduler kyber registered [ 3.431390] io scheduler bfq registered [ 3.433052] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.434668] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.436916] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.439033] ACPI: Power Button [PWRF] [ 3.443723] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.449606] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.490278] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.511144] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.531372] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.557263] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.585966] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.597939] Non-volatile memory driver v1.3 [ 3.599316] Linux agpgart interface v0.103 [ 3.630922] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.633587] vda: detected capacity change from 0 to 74686464 [ 3.658310] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.660611] vdb: detected capacity change from 0 to 1073741824 [ 3.683316] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.685332] vdc: detected capacity change from 0 to 2621440000 [ 3.704193] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.706198] vdd: detected capacity change from 0 to 2621440000 [ 3.728113] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.730469] vde: detected capacity change from 0 to 4294967296 [ 3.756354] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.758910] vdf: detected capacity change from 0 to 4294967296 [ 3.794940] libphy: Fixed MDIO Bus: probed [ 3.805120] usbcore: registered new interface driver usbserial_generic [ 3.807083] usbserial: USB Serial support registered for generic [ 3.809254] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.812887] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.814369] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.817634] mousedev: PS/2 mouse device common for all mice [ 3.820954] rtc_cmos 00:05: RTC can wake from S4 [ 3.824176] rtc_cmos 00:05: registered as rtc0 [ 3.825767] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.827127] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.828142] intel_pstate: CPU model not supported [ 3.833875] hid: raw HID events driver (C) Jiri Kosina [ 3.834378] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.836519] usbcore: registered new interface driver usbhid [ 3.841163] usbhid: USB HID core driver [ 3.842599] drop_monitor: Initializing network drop monitor service [ 3.844827] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.844787] Initializing XFRM netlink socket [ 3.849153] NET: Registered protocol family 10 [ 3.851553] Segment Routing with IPv6 [ 3.852525] NET: Registered protocol family 17 [ 3.854119] mpls_gso: MPLS GSO support [ 3.861123] RAS: Correctable Errors collector initialized. [ 3.862787] AVX version of gcm_enc/dec engaged. [ 3.864213] AES CTR mode by8 optimization enabled [ 3.966939] sched_clock: Marking stable (3966910556, 0)->(4869722604, -902812048) [ 3.970767] registered taskstats version 1 [ 3.973156] Loading compiled-in X.509 certificates [ 3.974480] zswap: loaded using pool lzo/zbud [ 4.005873] Key type big_key registered [ 4.018938] Key type encrypted registered [ 4.020714] ima: No TPM chip found, activating TPM-bypass! [ 4.022752] ima: Allocated hash algorithm: sha1 [ 4.024261] ima: No architecture policies found [ 4.025880] evm: Initialising EVM extended attributes: [ 4.027586] evm: security.selinux [ 4.028837] evm: security.ima [ 4.029603] evm: security.capability [ 4.030934] evm: HMAC attrs: 0x1 [ 4.033143] rtc_cmos 00:05: setting system clock to 2026-07-30 23:02:11 UTC (1785452531) [ 4.039089] debug: unmapping init [mem 0xffffffffbd003000-0xffffffffbd1fffff] [ 4.042074] debug: unmapping init [mem 0xffffffffbbd82000-0xffffffffbc058fff] [ 4.046055] Write protecting the kernel read-only data: 28672k [ 4.049151] debug: unmapping init [mem 0xffffffffba403000-0xffffffffba5fffff] [ 4.051874] debug: unmapping init [mem 0xffffffffbad14000-0xffffffffbadfffff] [ 4.083948] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.091237] systemd[1]: Detected virtualization kvm. [ 4.092624] systemd[1]: Detected architecture x86-64. [ 4.094121] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.120543] systemd[1]: No hostname configured. [ 4.122037] systemd[1]: Set hostname to . [ 4.123666] random: systemd: uninitialized urandom read (16 bytes read) [ 4.125717] systemd[1]: Initializing machine ID from random generator. [ 4.163521] random: ln: uninitialized urandom read (6 bytes read) [ 4.232454] random: systemd: uninitialized urandom read (16 bytes read) [ 4.234439] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.239322] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.243834] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.809981] device-mapper: uevent: version 1.0.3 [ 4.812152] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.497837] random: fast init done [ 5.508646] virtio_net virtio0 ens2: renamed from eth0 [ 5.553293] scsi host0: ata_piix [ 5.564383] scsi host1: ata_piix [ 5.565801] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.567744] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.251345] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.535256] random: crng init done [ 10.536528] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.833445] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.934943] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.194194] SELinux: Disabled at runtime. [ 12.258643] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.267488] systemd[1]: Detected virtualization kvm. [ 12.269379] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.749057] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.752225] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.756761] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.761247] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.764440] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.771561] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.776162] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK [[ 12.821711] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.241312] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.702086] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.759819] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.859628] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.891097] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (5s / no limit)[ 19.057377] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (6s / no limit) [*** ] A start job is running for Configur…-only root support (6s / no limit)[ 20.048964] NFS: Registering the id_resolver key type [ 20.054941] Key type id_resolver registered [ 20.060450] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (7s / no limit) [ *** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg221-server login: [ 76.694441] hrtimer: interrupt took 3277979 ns [ 100.376880] libcfs: loading out-of-tree module taints kernel. [ 100.689831] Key type ._llcrypt registered [ 100.697036] Key type .llcrypt registered [ 100.929217] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 132.867573] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 136.238104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 136.253981] alg: No test for adler32 (adler32-zlib) [ 137.969508] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 138.976420] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 140.831432] Key type lgssc registered [ 143.227637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 162.534364] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 225.036035] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 247.517884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 247.542076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 249.049390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 249.109809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 249.357905] Lustre: lustre-MDT0000: new disk, initializing [ 249.655537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 249.707313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 255.191733] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.665183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 275.950791] Lustre: 6534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 276.027336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 276.035084] Lustre: Skipped 1 previous similar message [ 276.203220] Lustre: lustre-MDT0001: new disk, initializing [ 276.318285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 276.396641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 276.414522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 283.699097] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 289.779413] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 303.799821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 304.428168] Lustre: lustre-OST0000: new disk, initializing [ 304.433918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 304.477830] Lustre: 8472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 304.650215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 313.940137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 313.953470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 314.198286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 315.648479] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 336.055143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 336.328133] Lustre: lustre-OST0001: new disk, initializing [ 336.338850] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 336.352978] Lustre: 9547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 336.544215] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 342.077459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 342.094944] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 342.170987] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 346.194963] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 361.595443] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 371.205611] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 379.824515] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing check_logdir /tmp/testlogs/ [ 388.447200] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing yml_node [ 395.182455] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 399.001186] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 402.234817] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 404.396951] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Thu Jul 30 19:08:50 EDT 2026 [ 428.862075] Lustre: DEBUG MARKER: - need MDS1_VERSION <= 2.16.61-1-g89cf292a8c2 (34682881 <= 34618625) for LU-18938, skip 360 [ 430.827938] Lustre: DEBUG MARKER: - need MDS1_VERSION < v2_14_55-100-g8a84c7f9c7 (34682881 < 34486116) for LU-14927, skip 0f [ 432.311478] Lustre: DEBUG MARKER: - need OST1_VERSION < v2_17_50-410-gbd8e7b9555 (34682881 < 34681754) for LU-12550, skip 216 [ 434.547757] Lustre: DEBUG MARKER: excepting tests: 42a 42c 42b 118c 118d 407 119i 216 817 411a [ 436.821526] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 51c 51e 834 [ 439.313692] Lustre: DEBUG MARKER: === sanity: start setup 19:09:24 (1785452964) === [ 446.981599] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing check_config_client /mnt/lustre [ 472.543673] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 477.625631] Lustre: 13558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 484.290216] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 492.153984] Lustre: DEBUG MARKER: === sanity: finish setup 19:10:17 (1785453017) === [ 500.745457] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 19:10:26 (1785453026) [ 506.049285] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 508.601877] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 19:10:34 (1785453034) [ 519.600563] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 19:10:45 (1785453045) [ 754.192874] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 19:14:39 (1785453279) [ 763.480046] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 19:14:49 (1785453289) [ 765.626833] Lustre: *** cfs_fail_loc=15b, val=0*** [ 765.628674] Lustre: *** cfs_fail_loc=15b, val=0*** [ 765.641414] LustreError: 6542:0:(llog_cat_server.c:483:llog_cat_add_rec()) lustre-OST0001-osc-MDT0000: initialization error: rc = -28 [ 775.052869] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 19:15:00 (1785453300) [ 783.888339] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 19:15:09 (1785453309) [ 785.328757] Lustre: *** cfs_fail_loc=19a, val=0*** [ 785.331216] ------------[ cut here ]------------ [ 785.338534] do not call blocking ops when !TASK_RUNNING; state=402 set at [<000000008ad8c18d>] distribute_txn_commit_thread+0x95/0x1000 [ptlrpc] [ 785.357320] WARNING: CPU: 2 PID: 7494 at kernel/sched/core.c:7471 __might_sleep+0x9d/0xc0 [ 785.368918] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 785.462278] CPU: 2 PID: 7494 Comm: dist_txn-1 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 785.479544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 785.496831] RIP: 0010:__might_sleep+0x9d/0xc0 [ 785.504273] Code: a8 12 00 00 48 c7 c7 a8 56 b8 ba 48 83 05 ba 89 df 02 01 c6 05 2f 83 52 02 01 48 89 d1 e8 a7 50 fb ff 48 83 05 ab 89 df 02 01 <0f> 0b 48 83 05 a9 89 df 02 01 48 83 05 a9 89 df 02 01 eb 9b 66 66 [ 785.534497] RSP: 0018:ffff9abd0793fd78 EFLAGS: 00010202 [ 785.547050] RAX: 0000000000000000 RBX: ffffffffbabadd7d RCX: 0000000000000000 [ 785.557779] RDX: ffff8a03c212e640 RSI: ffff8a03c211e5a8 RDI: ffff8a03c211e5a8 [ 785.568527] RBP: 00000000000000e2 R08: 0000000000000000 R09: c0000000ffff7fff [ 785.574983] R10: 0000000000000001 R11: ffff9abd0793fb68 R12: 0000000000000000 [ 785.582392] R13: 0000000000000001 R14: 0000000000000058 R15: ffffffffc0d15ce8 [ 785.590161] FS: 0000000000000000(0000) GS:ffff8a03c2100000(0000) knlGS:0000000000000000 [ 785.598983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 785.604972] CR2: 0000556a3b949e24 CR3: 0000000112a16004 CR4: 0000000000170ee0 [ 785.613486] Call Trace: [ 785.616186] ? show_regs.cold.9+0x22/0x2f [ 785.620438] ? __warn+0xc8/0x150 [ 785.624257] ? __might_sleep+0x9d/0xc0 [ 785.636154] ? report_bug+0x113/0x140 [ 785.640545] ? do_error_trap+0xb6/0x130 [ 785.646981] ? do_invalid_op+0x46/0x60 [ 785.654114] ? __might_sleep+0x9d/0xc0 [ 785.660243] ? invalid_op+0x14/0x20 [ 785.665840] ? distribute_txn_commit_batchid_update+0x68/0x8f0 [ptlrpc] [ 785.678100] ? __might_sleep+0x9d/0xc0 [ 785.682309] ? __might_sleep+0x95/0xc0 [ 785.691675] slab_pre_alloc_hook.constprop.64+0x11f/0x1d0 [ 785.699743] kmem_cache_alloc_trace+0x5b/0x440 [ 785.707809] distribute_txn_commit_batchid_update+0x68/0x8f0 [ptlrpc] [ 785.713665] distribute_txn_commit_thread+0xa97/0x1000 [ptlrpc] [ 785.728808] ? __schedule+0x369/0xcb0 [ 785.735445] ? do_raw_spin_unlock+0x75/0x190 [ 785.741278] ? distribute_txn_commit_batchid_update+0x8f0/0x8f0 [ptlrpc] [ 785.748885] kthread+0x1d1/0x200 [ 785.752327] ? set_kthread_struct+0x70/0x70 [ 785.757134] ret_from_fork+0x1f/0x30 [ 785.759718] ---[ end trace 10d2a060203c932d ]--- [ 786.699667] Lustre: *** cfs_fail_loc=19a, val=0*** [ 787.863558] Lustre: *** cfs_fail_loc=19a, val=0*** [ 790.640719] Lustre: *** cfs_fail_loc=19a, val=0*** [ 790.654081] Lustre: Skipped 1 previous similar message [ 795.497900] Lustre: *** cfs_fail_loc=19a, val=0*** [ 795.500741] Lustre: Skipped 3 previous similar messages [ 804.413116] Lustre: *** cfs_fail_loc=19a, val=0*** [ 804.414852] Lustre: Skipped 6 previous similar messages [ 821.204622] Lustre: *** cfs_fail_loc=19a, val=0*** [ 821.206144] Lustre: Skipped 14 previous similar messages [ 853.206617] Lustre: *** cfs_fail_loc=19a, val=0*** [ 853.208395] Lustre: Skipped 18 previous similar messages [ 930.177449] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 19:17:36 (1785453456) [ 931.475268] Lustre: *** cfs_fail_loc=188, val=0*** [ 935.084238] Lustre: *** cfs_fail_loc=189, val=0*** [ 946.130322] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 948.256749] Lustre: DEBUG MARKER: == sanity test 60j: llog_reader reports corruptions ====== 19:17:54 (1785453474) [ 956.774937] Lustre: lustre-MDD0000: changelog on [ 961.228549] Lustre: lustre-MDD0001: changelog on [ 974.597176] Lustre: DEBUG MARKER: SKIP: sanity test_60j path oi.1/0x1:0xc:0x0 is not in 'O/1/d/' format [ 978.588144] Lustre: lustre-MDD0001: changelog off [ 982.878730] Lustre: lustre-MDD0000: changelog off [ 986.544136] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 19:18:32 (1785453512) [ 997.197117] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 19:18:42 (1785453522) [ 1006.409411] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 19:18:51 (1785453531) [ 1078.587989] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 19:20:04 (1785453604) [ 1095.756706] Lustre: DEBUG MARKER: == sanity test 63c: test sync_on_close=1 ================= 19:20:21 (1785453621) [ 1121.529675] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 19:20:46 (1785453646) [ 1132.498792] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 1135.075460] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 19:21:00 (1785453660) [ 1147.448471] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 19:21:12 (1785453672) [ 1231.623725] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 19:22:37 (1785453757) [ 1240.044051] Lustre: *** cfs_fail_loc=725, val=0*** [ 1247.777110] Lustre: *** cfs_fail_loc=725, val=0*** [ 1258.873977] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 19:23:04 (1785453784) [ 1284.489263] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 19:23:29 (1785453809) [ 1318.613833] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 19:24:04 (1785453844) [ 1347.435935] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 19:24:33 (1785453873) [ 1363.435062] Lustre: *** cfs_fail_loc=513, val=17*** [ 1363.439191] LustreError: 14693:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1872182787556736 [ 1366.120034] Lustre: Failing over lustre-OST0000 [ 1366.467115] Lustre: server umount lustre-OST0000 complete [ 1369.057600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1373.337821] LustreError: 9749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1373.370984] LustreError: 9749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1374.185191] LustreError: 8455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.220913] LustreError: 8455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1378.446517] LustreError: 14705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.565507] LustreError: 8458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.598891] LustreError: 8458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1387.652369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1388.026812] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 1388.063931] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1388.682830] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1389.318934] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 1389.324958] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1396.215602] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [ 1409.461992] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1411.131397] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 1428.696654] Lustre: DEBUG MARKER: == sanity test 64j: check grants on re-done rpc ========== 19:25:54 (1785453954) [ 1430.443300] Lustre: *** cfs_fail_loc=256, val=0*** [ 1443.110499] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 19:26:08 (1785453968) [ 1452.572695] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 19:26:18 (1785453978) [ 1463.231954] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 19:26:28 (1785453988) [ 1475.278771] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 19:26:39 (1785453999) [ 1488.391996] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 19:26:52 (1785454012) [ 1505.554276] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 19:27:10 (1785454030) [ 1516.569962] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 19:27:21 (1785454041) [ 1529.417176] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 19:27:35 (1785454055) [ 1539.967991] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 19:27:45 (1785454065) [ 1553.916585] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 19:27:59 (1785454079) [ 1565.675592] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 19:28:10 (1785454090) [ 1568.633808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.656594] Lustre: Skipped 1 previous similar message [ 1568.673247] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 1568.677384] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1568.679252] Lustre: Skipped 1 previous similar message [ 1570.045230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.078319] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1570.092372] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1571.470666] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 1572.709504] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1572.720374] Lustre: Skipped 1 previous similar message [ 1572.733343] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1572.748324] Lustre: Skipped 1 previous similar message [ 1593.382942] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1616.875505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1616.910101] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 1616.913276] Lustre: Skipped 1 previous similar message [ 1616.931771] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1616.965147] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1625.681563] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1626.093089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1634.013916] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1634.374737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1660.678660] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1680.266987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.293244] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1680.302074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1680.315466] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1687.481882] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1687.771400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1694.767487] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1695.065323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1721.939506] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1746.032905] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.060777] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 1746.064215] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1746.077830] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1753.503712] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1753.798727] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1760.671478] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1760.987541] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1784.647508] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1811.753321] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1811.766284] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1811.776294] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1811.789451] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1819.540054] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1820.003657] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1826.414792] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1826.771445] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1834.957219] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 19:32:40 (1785454360) [ 1845.002949] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 19:32:50 (1785454370) [ 1855.943389] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 19:33:00 (1785454380) [ 1897.263518] Lustre: DEBUG MARKER: == sanity test 65o: pool inheritance for mdt component === 19:33:43 (1785454423) [ 1938.556356] Lustre: DEBUG MARKER: == sanity test 65p: setstripe with yaml file and huge number ========================================================== 19:34:23 (1785454463) [ 1952.233199] Lustre: DEBUG MARKER: == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 19:34:38 (1785454478) [ 1962.782718] Lustre: DEBUG MARKER: == sanity test 65r: prevent all-zero offsets ============= 19:34:48 (1785454488) [ 1972.622507] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 19:34:58 (1785454498) [ 1998.336964] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 19:35:23 (1785454523) [ 2001.929260] Lustre: *** cfs_fail_loc=217, val=0*** [ 2001.933643] Lustre: Skipped 1 previous similar message [ 2006.020151] Lustre: *** cfs_fail_loc=217, val=0*** [ 2017.722722] Lustre: DEBUG MARKER: == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 19:35:43 (1785454543) [ 2036.248931] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 2039.331504] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 19:36:04 (1785454564) [ 2048.117915] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 19:36:14 (1785454574) [ 2061.235514] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 19:36:25 (1785454585) [ 2098.833811] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 19:37:04 (1785454624) [ 2108.084873] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 19:37:13 (1785454633) [ 2116.563768] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 19:37:22 (1785454642) [ 2126.341757] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 19:37:31 (1785454651) [ 2252.381343] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 19:39:37 (1785454777) [ 2329.770586] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 19:40:55 (1785454855) [ 2343.131728] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 19:41:07 (1785454867) [ 2345.066976] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.21@tcp inode [0x200000407:0xc1d:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum 73edb9f9, server csum 73edb9f8 [ 2348.019734] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 2349.909609] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.21@tcp inode [0x200000407:0xc1d:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum e062897e (type 1), server csum 5448f3a0 (type 1) [ 2354.714247] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 2356.440622] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.21@tcp inode [0x200000407:0xc1d:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 26ff1f5e (type 2), server csum 33f21fa6 (type 2) [ 2360.506117] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 2361.500147] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.21@tcp inode [0x200000407:0xc1d:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum d69db7b9 (type 4), server csum 73edb9f8 (type 4) [ 2365.711477] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 2367.198594] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.21@tcp inode [0x200000407:0xc1d:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 2d54b5ba (type 10), server csum efc2b671 (type 10) [ 2372.431588] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2374.056436] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.21@tcp inode [0x200000407:0xc1d:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 5c9bfaeb (type 20), server csum 1c47faa3 (type 20) [ 2378.178403] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 2384.091547] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 2385.828107] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.21@tcp inode [0x200000407:0xc1d:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 8e0ef923 (type 80), server csum 7dfaf8c1 (type 80) [ 2385.844220] LustreError: Skipped 1 previous similar message [ 2390.672235] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 2400.543210] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 19:42:06 (1785454926) [ 2415.250304] Lustre: 8462:0:(tgt_handler.c:2012:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc1e:0x0]:[0-1048575]-daaccd1d-477039f7 [ 2415.279481] LustreError: dumping log to /tmp/lustre-log.1785454942.8462 [ 2415.511131] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.21@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [0-1048575], client returned csum daaccd1d (type 4), server csum 477039f7 (type 4) [ 2466.091297] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 19:43:11 (1785454991) [ 2466.784357] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.21@tcp inode [0x200000407:0xc20:0x0] object 0x280000401:3839 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 2469.411639] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.21@tcp inode [0x200000407:0xc20:0x0] object 0x280000401:3839 extent [0-4194303], client returned csum f5a99216 (type 4), server csum de2bf73e (type 4) [ 2478.599636] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 19:43:24 (1785455004) [ 2481.496304] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 2482.095351] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.21@tcp inode [0x200000407:0xc21:0x0] object 0x2c0000401:3831 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 2486.626932] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 2487.066456] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.21@tcp inode [0x200000407:0xc22:0x0] object 0x280000401:3840 extent [0-4194303]: client csum 7d33b9a0, server csum 7d33b99f [ 2487.087580] LustreError: Skipped 3 previous similar messages [ 2490.514503] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 2492.151313] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.21@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3832 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 2492.172772] LustreError: Skipped 4 previous similar messages [ 2494.866708] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 2499.539388] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2500.291042] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.21@tcp inode [0x200000407:0xc25:0x0] object 0x2c0000401:3833 extent [0-4194303]: client csum fec05011, server csum fec05010 [ 2500.315225] LustreError: Skipped 5 previous similar messages [ 2503.805903] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 2508.429327] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 2520.711859] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 2523.661059] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 19:44:08 (1785455048) [ 2526.653054] Lustre: *** cfs_fail_loc=21a, val=0*** [ 2526.655269] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.21@tcp inode [0x200000407:0xc28:0x0] object 0x280000401:3843 extent [0-1048575]: client csum 477039f7, server csum 4d764975 [ 2526.698776] LustreError: Skipped 11 previous similar messages [ 2534.187910] Lustre: *** cfs_fail_loc=21b, val=0*** [ 2546.994548] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 19:44:31 (1785455071) [ 2548.517980] Lustre: Setting parameter lustre.osc.lustre*.checksums=0 in log params [ 2557.723135] Lustre: Modifying parameter lustre.osc.lustre*.checksums=1 in log params [ 2562.262194] Lustre: Disabling parameter lustre.osc.lustre*.checksums= in log params [ 2587.131867] Lustre: Setting parameter lustre.osc.lustre*.checksums=0 in log params [ 2592.095065] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 19:45:17 (1785455117) [ 2594.830099] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 2596.980187] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 2604.828527] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2607.179582] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in IDLE state after 0 sec [ 2616.817475] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2619.026376] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in FULL state after 0 sec [ 2622.053783] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 2633.171491] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2636.213632] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in IDLE state after 0 sec [ 2646.201575] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2647.981465] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in FULL state after 0 sec [ 2650.440373] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 2658.151377] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2664.572931] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in IDLE state after 3 sec [ 2672.165429] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2673.948458] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in FULL state after 0 sec [ 2676.414920] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 2683.764368] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2689.182991] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in IDLE state after 3 sec [ 2697.324282] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2699.230104] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in FULL state after 0 sec [ 2701.591535] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2709.542551] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2714.519983] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in IDLE state after 3 sec [ 2721.492237] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2723.526564] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in FULL state after 0 sec [ 2726.331788] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 2735.145698] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2740.633882] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in IDLE state after 3 sec [ 2749.874315] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2752.029314] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in FULL state after 0 sec [ 2754.735186] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 2765.681961] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2771.777611] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in IDLE state after 3 sec [ 2780.358990] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid 50 [ 2782.547132] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff999486cd0000.ost_server_uuid in FULL state after 0 sec [ 2792.846530] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 2795.544365] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 19:48:40 (1785455320) [ 2804.489641] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 19:48:49 (1785455329) [ 2807.661667] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 2810.262253] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2812.299264] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 2814.239322] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 2821.263451] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 2823.114451] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 19:49:09 (1785455349) [ 2838.576946] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 19:49:24 (1785455364) [ 2852.346865] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check =========== 19:49:37 (1785455377) [ 2874.015433] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 19:49:59 (1785455399) [ 2885.676825] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 19:50:10 (1785455410) [ 2887.577297] Lustre: *** cfs_fail_loc=228, val=0*** [ 2897.178734] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 19:50:22 (1785455422) [ 2899.227354] Lustre: *** cfs_fail_loc=228, val=0*** [ 2909.466316] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 19:50:34 (1785455434) [ 2940.957883] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ========================================================== 19:51:06 (1785455466) [ 2954.229150] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 19:51:18 (1785455478) [ 3173.933619] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 19:54:59 (1785455699) [ 3223.854805] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 19:55:49 (1785455749) [ 3349.133314] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 19:57:54 (1785455874) [ 3767.496113] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 20:04:52 (1785456292) [ 3895.920483] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 20:07:01 (1785456421) [ 3908.160579] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 20:07:14 (1785456434) [ 4007.099894] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 20:08:52 (1785456532) [ 4038.096567] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 20:09:23 (1785456563) [ 4050.488458] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 20:09:36 (1785456576) [ 4138.462573] Lustre: DEBUG MARKER: == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 20:11:04 (1785456664) [ 4156.507637] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 20:11:21 (1785456681) [ 4168.166331] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 20:11:33 (1785456693) [ 4192.099230] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 20:11:57 (1785456717) [ 4203.280952] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 20:12:09 (1785456729) [ 4224.753479] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 20:12:30 (1785456750) [ 4246.345686] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 20:12:52 (1785456772) [ 4249.506568] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 4251.582196] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 4253.634744] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 4255.921676] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 4264.971184] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 20:13:10 (1785456790) [ 4269.663847] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 4272.475143] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 4274.858505] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 4276.761785] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 4279.518946] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 4288.028958] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 20:13:34 (1785456814) [ 4298.568181] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 20:13:43 (1785456823) [ 4323.113787] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 20:14:08 (1785456848) [ 4333.463440] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 20:14:19 (1785456859) [ 4344.769415] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 20:14:30 (1785456870) [ 4356.465928] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 20:14:41 (1785456881) [ 4371.349344] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 20:14:56 (1785456896) [ 4381.466744] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 20:15:06 (1785456906) [ 4392.058975] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 20:15:17 (1785456917) [ 4402.055385] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 20:15:27 (1785456927) [ 4411.621897] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 20:15:37 (1785456937) [ 4422.478320] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 20:15:48 (1785456948) [ 4792.317524] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 20:21:58 (1785457318) [ 5014.804838] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 20:25:39 (1785457539) [ 5025.836742] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 20:25:50 (1785457550) [ 5052.145701] Lustre: 60447:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 30549 > trans_max 3200 [ 5052.153963] Lustre: 60447:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 5052.162090] Lustre: 60447:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [ 5052.171375] Lustre: 60447:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/142/0, punch: 0/0/0, quota 0/0/0 [ 5052.179300] Lustre: 60447:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 5052.185385] Lustre: 60447:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 5052.192275] CPU: 2 PID: 60447 Comm: mdt00_006 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 5052.198484] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5052.204308] Call Trace: [ 5052.206244] ? dump_stack+0xbb/0x10e [ 5052.208412] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [ 5052.211421] ? dt_trans_start+0x1c/0x70 [obdclass] [ 5052.214169] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [ 5052.217399] ? lod_declare_ref_add+0x1a/0x30 [lod] [ 5052.221123] ? dt_declare_ref_add+0x6c/0x200 [obdclass] [ 5052.224553] ? lod_trans_start+0x109/0x4c0 [lod] [ 5052.227905] ? mdd_declare_links_add+0x60/0x90 [mdd] [ 5052.231017] ? mdd_trans_start+0x18/0x30 [mdd] [ 5052.234976] ? mdd_unlink+0x7a4/0x13c0 [mdd] [ 5052.238368] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [ 5052.242168] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 5052.244917] ? mdt_reint_internal+0x693/0xdc0 [mdt] [ 5052.248945] ? mdt_reint+0x163/0x190 [mdt] [ 5052.251368] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 5052.255188] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [ 5052.259365] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 5052.264661] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5052.269105] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 5052.272370] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [ 5052.276203] ? kthread+0x1d1/0x200 [ 5052.278582] ? set_kthread_struct+0x70/0x70 [ 5052.280906] ? ret_from_fork+0x1f/0x30 [ 5063.031491] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 20:26:28 (1785457588) [ 5070.858291] Lustre: lustre-MDD0000: changelog on [ 5074.594217] Lustre: lustre-MDD0001: changelog on [ 5086.210340] Lustre: lustre-MDD0001: changelog off [ 5090.306419] Lustre: lustre-MDD0000: changelog off [ 5093.772504] Lustre: DEBUG MARKER: == sanity test 103g: no MDS_GETXATTR storm for inodes without ACL_ACCESS (LU-17238) ========================================================== 20:26:59 (1785457619) [ 5102.050655] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 20:27:08 (1785457628) [ 5102.726361] Lustre: lustre-OST0000: Client 4c222f6d-6e24-4975-838e-bb9ff291fc54 (at 192.168.202.21@tcp) reconnecting [ 5109.375933] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99939a618000.ost_server_uuid 50 [ 5111.232222] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff99939a618000.ost_server_uuid in FULL state after 0 sec [ 5121.317600] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 20:27:27 (1785457647) [ 5131.587961] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 20:27:36 (1785457656) [ 5133.479384] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 5135.624658] Lustre: DEBUG MARKER: == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 20:27:41 (1785457661) [ 5143.314293] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 20:27:49 (1785457669) [ 5154.016059] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 20:27:58 (1785457678) [ 5163.356635] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 20:28:09 (1785457689) [ 5172.478607] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 20:28:18 (1785457698) [ 5191.361625] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 20:28:37 (1785457717) [ 5202.174125] Lustre: DEBUG MARKER: == sanity test 105f: Enqueue same range flocks =========== 20:28:47 (1785457727) [ 5215.473595] Lustre: DEBUG MARKER: == sanity test 105g: ldlm_lock_debug stack test ========== 20:29:00 (1785457740) [ 5227.193798] Lustre: DEBUG MARKER: == sanity test 105h: Flock functional verify ============= 20:29:12 (1785457752) [ 5237.371896] Lustre: DEBUG MARKER: == sanity test 105i: Flock deadlock verify =============== 20:29:22 (1785457762) [ 5253.385062] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 20:29:38 (1785457778) [ 5269.722976] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 20:29:54 (1785457794) [ 5283.003539] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 20:30:08 (1785457808) [ 5293.673341] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 20:30:19 (1785457819) [ 5496.968227] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 20:33:42 (1785458022) [ 5500.580949] Lustre: *** cfs_fail_loc=147, val=0*** [ 5501.090749] Lustre: *** cfs_fail_loc=147, val=0*** [ 5501.097218] Lustre: Skipped 25 previous similar messages [ 5516.508603] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 20:34:00 (1785458040) [ 5526.764758] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 20:34:12 (1785458052) [ 5537.815908] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 20:34:23 (1785458063) [ 5541.013543] Lustre: *** cfs_fail_loc=217, val=0*** [ 5541.025877] Lustre: Skipped 51 previous similar messages [ 5554.259073] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 5557.415623] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 5560.374513] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 20:34:45 (1785458085) [ 5574.879899] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 20:34:59 (1785458099) [ 5585.671534] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 20:35:11 (1785458111) [ 5588.454193] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5589.520666] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5591.566184] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5595.154128] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5610.111840] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 20:35:35 (1785458135) [ 5613.803986] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5613.808716] Lustre: Skipped 1 previous similar message [ 5633.549979] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 20:35:59 (1785458159) [ 5637.169709] Lustre: *** cfs_fail_loc=220, val=0*** [ 5637.175163] Lustre: Skipped 3 previous similar messages [ 5648.507491] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 20:36:13 (1785458173) [ 5653.505469] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5653.507554] Lustre: Skipped 1 previous similar message [ 5680.497437] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 20:36:45 (1785458205) [ 5693.842720] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 20:36:59 (1785458219) [ 5704.794077] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 20:37:10 (1785458230) [ 5709.279687] LustreError: 8460:0:(ofd_dev.c:1917:ofd_statfs_hdl()) cfs_fail_timeout id 242 sleeping for 10000ms [ 5709.794086] LustreError: 11462:0:(ofd_dev.c:1917:ofd_statfs_hdl()) cfs_fail_timeout id 242 sleeping for 10000ms [ 5710.760235] LustreError: 8460:0:(ofd_dev.c:1917:ofd_statfs_hdl()) cfs_fail_timeout interrupted [ 5710.771720] LustreError: 8460:0:(ofd_dev.c:1917:ofd_statfs_hdl()) Skipped 1 previous similar message [ 5719.820947] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 20:37:25 (1785458245) [ 5728.702991] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 20:37:34 (1785458254) [ 5739.626465] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 20:37:44 (1785458264) [ 5749.152606] Lustre: DEBUG MARKER: == sanity test 119e: Basic tests of dio read and write at various sizes ========================================================== 20:37:54 (1785458274) [ 5795.955861] Lustre: DEBUG MARKER: == sanity test 119f: dio vs dio race ===================== 20:38:41 (1785458321) [ 5856.914847] Lustre: DEBUG MARKER: == sanity test 119g: dio vs buffered I/O race ============ 20:39:42 (1785458382) [ 5911.262830] Lustre: DEBUG MARKER: == sanity test 119h: basic tests of memory unaligned dio ========================================================== 20:40:36 (1785458436) [ 5955.267586] Lustre: DEBUG MARKER: SKIP: sanity test_119i skipping ALWAYS excluded test 119i [ 5957.413500] Lustre: DEBUG MARKER: == sanity test 119j: basic tests of hybrid IO switching == 20:41:23 (1785458483) [ 5966.282731] Lustre: DEBUG MARKER: == sanity test 119k: hybrid IO counting with stats and disabling ========================================================== 20:41:31 (1785458491) [ 5994.064556] Lustre: DEBUG MARKER: == sanity test 119m: Test DIO readv/writev: exercise iter duplication ========================================================== 20:41:58 (1785458518) [ 6003.423137] Lustre: DEBUG MARKER: == sanity test 119n: Test Unaligned DIO readv() and writev() with unpatched ZFS ========================================================== 20:42:08 (1785458528) [ 6005.032573] Lustre: DEBUG MARKER: SKIP: sanity test_119n need ZFS server without unaligned_dio support [ 6008.490030] Lustre: DEBUG MARKER: == sanity test 119o: Test Unaligned DIO readv() and writev() with unpatched servers ========================================================== 20:42:13 (1785458533) [ 6010.668389] Lustre: DEBUG MARKER: SKIP: sanity test_119o need ldiskfs without unaligned_dio support. [ 6013.278550] Lustre: DEBUG MARKER: == sanity test 119p: Test Unaligned DIO readv() and writev() with patched servers ========================================================== 20:42:18 (1785458538) [ 6023.615552] Lustre: DEBUG MARKER: == sanity test 119q: Test patchded Unaligned DIO readv() and writev() ========================================================== 20:42:29 (1785458549) [ 6042.687904] Lustre: DEBUG MARKER: == sanity test 119r: Test error handling in unaligned DIO user copy ========================================================== 20:42:48 (1785458568) [ 6055.693202] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 20:43:00 (1785458580) [ 6071.262459] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 20:43:16 (1785458596) [ 6083.566719] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 20:43:29 (1785458609) [ 6100.802593] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 20:43:45 (1785458625) [ 6125.445742] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 20:44:10 (1785458650) [ 6145.682054] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 20:44:31 (1785458671) [ 6169.906411] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 20:44:54 (1785458694) [ 6706.972853] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 20:53:51 (1785459231) [ 6717.660445] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 20:54:03 (1785459243) [ 6727.347703] Lustre: DEBUG MARKER: 'ls -l' 100 files without statahead: 2 sec [ 6731.137253] Lustre: DEBUG MARKER: 'ls -l' 100 files with statahead: 1 sec [ 6781.774467] Lustre: DEBUG MARKER: 'ls -l' 1000 files without statahead: 28 sec [ 6792.157142] Lustre: DEBUG MARKER: 'ls -l' 1000 files with statahead: 7 sec [ 6793.736778] Lustre: DEBUG MARKER: 'ls -l' done [ 6818.199951] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123aa.sanity: 22 seconds [ 6834.379823] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 20:55:59 (1785459359) [ 6847.949105] Lustre: DEBUG MARKER: 'statx -l' 100 files without statahead: 3 sec [ 6852.127586] Lustre: DEBUG MARKER: 'statx -l' 100 files with statahead: 1 sec [ 6909.263863] Lustre: DEBUG MARKER: 'statx -l' 1000 files without statahead: 31 sec [ 6922.427690] Lustre: DEBUG MARKER: 'statx -l' 1000 files with statahead: 9 sec [ 6924.782868] Lustre: DEBUG MARKER: 'statx -l' done [ 6953.811227] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ab.sanity: 25 seconds [ 6969.495964] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 20:58:15 (1785459495) [ 6982.291502] Lustre: DEBUG MARKER: 'statx -c 1 [ 6986.468871] Lustre: DEBUG MARKER: 'statx -c 1 [ 7042.024406] Lustre: DEBUG MARKER: 'statx -c 1 [ 7050.843620] Lustre: DEBUG MARKER: 'statx -c 1 [ 7052.683762] Lustre: DEBUG MARKER: 'statx -c 1 [ 7079.071605] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ac.sanity: 24 seconds [ 7089.169610] Lustre: DEBUG MARKER: 'statx --cached=always -D' 100 files without statahead: 0 sec [ 7091.995775] Lustre: DEBUG MARKER: 'statx --cached=always -D' 100 files with statahead: 0 sec [ 7116.020344] Lustre: DEBUG MARKER: 'statx --cached=always -D' 1000 files without statahead: 0 sec [ 7118.149559] Lustre: DEBUG MARKER: 'statx --cached=always -D' 1000 files with statahead: 1 sec [ 7278.978890] Lustre: DEBUG MARKER: 'statx --cached=always -D' 10000 files without statahead: 1 sec [ 7282.457962] Lustre: DEBUG MARKER: 'statx --cached=always -D' 10000 files with statahead: 1 sec [ 7285.522439] Lustre: DEBUG MARKER: 'statx --cached=always -D' done [ 7690.240362] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ac.sanity: 400 seconds [ 7711.546170] Lustre: DEBUG MARKER: == sanity test 123ad: Verify batching statahead works correctly ========================================================== 21:10:37 (1785460237) [ 7724.921254] Lustre: DEBUG MARKER: 'ls -l' 100 files without statahead: 3 sec [ 7728.894515] Lustre: DEBUG MARKER: 'ls -l' 100 files with statahead: 1 sec [ 7795.937550] Lustre: DEBUG MARKER: 'ls -l' 1000 files without statahead: 41 sec [ 7813.615362] Lustre: DEBUG MARKER: 'ls -l' 1000 files with statahead: 12 sec [ 7815.631511] Lustre: DEBUG MARKER: 'ls -l' done [ 7849.585626] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ad.sanity: 30 seconds [ 8656.957243] Lustre: DEBUG MARKER: 'ls -l' 100 files without statahead: 2 sec [ 8659.687642] Lustre: DEBUG MARKER: 'ls -l' 100 files with statahead: 0 sec [ 8711.212927] Lustre: DEBUG MARKER: 'ls -l' 1000 files without statahead: 26 sec [ 8725.752970] Lustre: DEBUG MARKER: 'ls -l' 1000 files with statahead: 9 sec [ 8727.760850] Lustre: DEBUG MARKER: 'ls -l' done [ 8756.245923] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds [ 9424.491287] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 21:39:10 (1785461950) [ 9453.688859] Lustre: DEBUG MARKER: ls done [ 9488.428538] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 21:40:13 (1785462013) [ 9502.582091] Lustre: DEBUG MARKER: == sanity test 123d: Statahead on striped directories works correctly ========================================================== 21:40:28 (1785462028) [ 9535.337570] Lustre: DEBUG MARKER: == sanity test 123e: statahead with large wide striping == 21:41:00 (1785462060) [ 9743.763694] Lustre: 82646:0:(mdt_handler.c:4704:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 9978.825673] Lustre: DEBUG MARKER: == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 21:48:20 (1785462500) [10180.609673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10185.232636] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10626.979578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10629.145981] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11407.974201] Lustre: 6542:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 26019 > trans_max 3200 [11407.997930] Lustre: 6542:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [11408.003623] Lustre: 6542:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1701/1701/0, xattr_set: 2551/24064/0 [11408.021534] Lustre: 6542:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/142/0, punch: 0/0/0, quota 0/0/0 [11408.028747] Lustre: 6542:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [11408.036220] Lustre: 6542:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [11408.047489] CPU: 3 PID: 6542 Comm: mdt00_002 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [11408.057434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11408.070086] Call Trace: [11408.075613] ? dump_stack+0xbb/0x10e [11408.083228] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [11408.096328] ? dt_trans_start+0x1c/0x70 [obdclass] [11408.108553] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [11408.111516] ? lod_declare_ref_add+0x1a/0x30 [lod] [11408.118434] ? dt_declare_ref_add+0x6c/0x200 [obdclass] [11408.128584] ? lod_trans_start+0x109/0x4c0 [lod] [11408.136526] ? mdd_declare_links_add+0x60/0x90 [mdd] [11408.154816] ? mdd_trans_start+0x18/0x30 [mdd] [11408.156221] ? mdd_unlink+0x7a4/0x13c0 [mdd] [11408.170750] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [11408.172300] ? mdt_reint_rec+0x139/0x2b0 [mdt] [11408.173675] ? mdt_reint_internal+0x693/0xdc0 [mdt] [11408.194640] ? mdt_reint+0x163/0x190 [mdt] [11408.205330] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [11408.220402] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [11408.230718] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [11408.247018] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11408.252138] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [11408.267117] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [11408.282751] ? kthread+0x1d1/0x200 [11408.288990] ? set_kthread_struct+0x70/0x70 [11408.298304] ? ret_from_fork+0x1f/0x30 [11469.710829] Lustre: 8773:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 30549 > trans_max 3200 [11469.725289] Lustre: 8773:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [11469.733955] Lustre: 8773:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [11469.751666] Lustre: 8773:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/142/0, punch: 0/0/0, quota 0/0/0 [11469.765620] Lustre: 8773:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [11469.789219] Lustre: 8773:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [11469.807272] CPU: 1 PID: 8773 Comm: mdt00_003 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [11469.830186] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11469.842735] Call Trace: [11469.843493] ? dump_stack+0xbb/0x10e [11469.844676] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [11469.867043] ? dt_trans_start+0x1c/0x70 [obdclass] [11469.893923] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [11469.901745] ? lod_declare_ref_add+0x1a/0x30 [lod] [11469.907901] ? dt_declare_ref_add+0x6c/0x200 [obdclass] [11469.916620] ? lod_trans_start+0x109/0x4c0 [lod] [11469.917647] ? mdd_declare_links_add+0x60/0x90 [mdd] [11469.918733] ? mdd_trans_start+0x18/0x30 [mdd] [11469.924509] ? mdd_unlink+0x7a4/0x13c0 [mdd] [11469.925846] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [11469.931154] ? mdt_reint_rec+0x139/0x2b0 [mdt] [11469.932975] ? mdt_reint_internal+0x693/0xdc0 [mdt] [11469.934377] ? mdt_reint+0x163/0x190 [mdt] [11469.942505] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [11469.944189] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [11469.950394] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [11469.953390] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11469.965325] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [11469.968885] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [11469.973841] ? kthread+0x1d1/0x200 [11469.974633] ? set_kthread_struct+0x70/0x70 [11469.975644] ? ret_from_fork+0x1f/0x30 [11578.102912] Lustre: 60460:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 30573 > trans_max 3200 [11578.117410] Lustre: 60460:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [11578.129293] Lustre: 60460:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [11578.137956] Lustre: 60460:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/166/0, punch: 0/0/0, quota 0/0/0 [11578.153957] Lustre: 60460:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [11578.162285] Lustre: 60460:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [11578.177127] CPU: 3 PID: 60460 Comm: mdt00_007 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [11578.195375] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11578.206596] Call Trace: [11578.207678] ? dump_stack+0xbb/0x10e [11578.208937] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [11578.225003] ? dt_trans_start+0x1c/0x70 [obdclass] [11578.234989] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [11578.244954] ? lod_declare_ref_add+0x1a/0x30 [lod] [11578.254016] ? dt_declare_ref_add+0x6c/0x200 [obdclass] [11578.262585] ? lod_trans_start+0x109/0x4c0 [lod] [11578.270381] ? mdd_declare_links_add+0x60/0x90 [mdd] [11578.281017] ? mdd_trans_start+0x18/0x30 [mdd] [11578.290730] ? mdd_unlink+0x7a4/0x13c0 [mdd] [11578.301085] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [11578.308765] ? mdt_reint_rec+0x139/0x2b0 [mdt] [11578.313445] ? mdt_reint_internal+0x693/0xdc0 [mdt] [11578.320954] ? mdt_reint+0x163/0x190 [mdt] [11578.331052] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [11578.340269] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [11578.350226] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [11578.357342] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11578.366584] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [11578.377823] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [11578.382140] ? kthread+0x1d1/0x200 [11578.389853] ? set_kthread_struct+0x70/0x70 [11578.402097] ? ret_from_fork+0x1f/0x30 [11639.259831] Lustre: 6542:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 30549 > trans_max 3200 [11639.279045] Lustre: 6542:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [11639.290567] Lustre: 6542:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [11639.304313] Lustre: 6542:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/142/0, punch: 0/0/0, quota 0/0/0 [11639.318091] Lustre: 6542:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [11639.332065] Lustre: 6542:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [11639.340922] CPU: 2 PID: 6542 Comm: mdt00_002 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [11639.354378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11639.363222] Call Trace: [11639.366327] ? dump_stack+0xbb/0x10e [11639.370448] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [11639.376608] ? dt_trans_start+0x1c/0x70 [obdclass] [11639.380842] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [11639.388845] ? lod_declare_ref_add+0x1a/0x30 [lod] [11639.394871] ? dt_declare_ref_add+0x6c/0x200 [obdclass] [11639.401563] ? lod_trans_start+0x109/0x4c0 [lod] [11639.404240] ? mdd_declare_links_add+0x60/0x90 [mdd] [11639.410138] ? mdd_trans_start+0x18/0x30 [mdd] [11639.419142] ? mdd_unlink+0x7a4/0x13c0 [mdd] [11639.424503] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [11639.428513] ? mdt_reint_rec+0x139/0x2b0 [mdt] [11639.438462] ? mdt_reint_internal+0x693/0xdc0 [mdt] [11639.447130] ? mdt_reint+0x163/0x190 [mdt] [11639.452480] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [11639.461248] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [11639.468530] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [11639.473671] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11639.481229] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [11639.484139] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [11639.487808] ? kthread+0x1d1/0x200 [11639.491661] ? set_kthread_struct+0x70/0x70 [11639.493936] ? ret_from_fork+0x1f/0x30 [11725.671452] Lustre: 6541:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 30573 > trans_max 3200 [11725.685746] Lustre: 6541:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [11725.694880] Lustre: 6541:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [11725.704647] Lustre: 6541:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/166/0, punch: 0/0/0, quota 0/0/0 [11725.712293] Lustre: 6541:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [11725.729040] Lustre: 6541:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [11725.738872] CPU: 1 PID: 6541 Comm: mdt00_001 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [11725.751755] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11725.769128] Call Trace: [11725.775978] ? dump_stack+0xbb/0x10e [11725.777254] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [11725.786610] ? dt_trans_start+0x1c/0x70 [obdclass] [11725.788288] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [11725.796933] ? lod_declare_ref_add+0x1a/0x30 [lod] [11725.798682] ? dt_declare_ref_add+0x6c/0x200 [obdclass] [11725.809631] ? lod_trans_start+0x109/0x4c0 [lod] [11725.810763] ? mdd_declare_links_add+0x60/0x90 [mdd] [11725.814291] ? mdd_trans_start+0x18/0x30 [mdd] [11725.819249] ? mdd_unlink+0x7a4/0x13c0 [mdd] [11725.827833] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [11725.830596] ? mdt_reint_rec+0x139/0x2b0 [mdt] [11725.835060] ? mdt_reint_internal+0x693/0xdc0 [mdt] [11725.838033] ? mdt_reint+0x163/0x190 [mdt] [11725.846957] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [11725.852004] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [11725.856417] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [11725.860928] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11725.863358] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [11725.866682] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [11725.873344] ? kthread+0x1d1/0x200 [11725.876155] ? set_kthread_struct+0x70/0x70 [11725.879190] ? ret_from_fork+0x1f/0x30 [11772.136142] Lustre: DEBUG MARKER: == sanity test 123g: Test for stat-ahead advise ========== 22:18:12 (1785464292) [11878.477865] Lustre: DEBUG MARKER: == sanity test 123h: Verify statahead work with the fname pattern via du ========================================================== 22:19:59 (1785464399) [14331.084468] Lustre: DEBUG MARKER: == sanity test 123i: Verify statahead work with the fname indexing pattern ========================================================== 23:00:55 (1785466855) [14484.993572] Lustre: DEBUG MARKER: == sanity test 123j: -ENOENT error from batched statahead be handled correctly ========================================================== 23:03:30 (1785467010) [14495.499636] Lustre: DEBUG MARKER: == sanity test 123k: Verify statahead work with mdtest shared stat() mode ========================================================== 23:03:40 (1785467020) [14498.506683] Lustre: DEBUG MARKER: SKIP: sanity test_123k mdtest not found [14500.987164] Lustre: DEBUG MARKER: == sanity test 123l: Avoid panic when revalidate a local cached entry ========================================================== 23:03:46 (1785467026) [14587.139583] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 23:05:12 (1785467112) [14590.099894] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [14645.562613] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff99948918d000 [14648.147943] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff99948918d000 [14650.526457] Lustre: DEBUG MARKER: LRU=1015 [14653.174793] Lustre: DEBUG MARKER: LIMIT=46162 [14655.458055] Lustre: DEBUG MARKER: LVF=5457500 [14657.937101] Lustre: DEBUG MARKER: OLD_LVF=100 [14661.077829] Lustre: DEBUG MARKER: Sleep 50 sec [14715.052981] Lustre: DEBUG MARKER: Dropped 580 locks in 50s [14717.803863] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [14765.634382] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 23:08:11 (1785467291) [14905.269722] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [15054.500795] Lustre: DEBUG MARKER: ls -la time: 144 seconds [15057.061043] Lustre: DEBUG MARKER: lru_size = 400 [15334.882847] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [15457.975441] Lustre: DEBUG MARKER: ls -la time: 115 seconds [15460.322729] Lustre: DEBUG MARKER: lru_size = 4052 [15463.011292] Lustre: DEBUG MARKER: ls -la is 20% faster with lru resize enabled [15567.697881] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 23:21:33 (1785468093) [15604.671971] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize disabled ========================================================== 23:22:10 (1785468130) [15643.025425] Lustre: DEBUG MARKER: == sanity test 124e: LFRU keep priv locks from eviction == 23:22:48 (1785468168) [15712.107743] Lustre: DEBUG MARKER: == sanity test 124f: LFRU priv threshold inc/dec adjustment ========================================================== 23:23:57 (1785468237) [15846.189942] Lustre: DEBUG MARKER: == sanity test 124g: LFRU performance test =============== 23:26:10 (1785468370) [17187.457464] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 23:48:32 (1785469712) [17199.511449] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 23:48:44 (1785469724) [17209.217812] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 23:48:54 (1785469734) [17219.900240] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 23:49:06 (1785469746) [17229.013626] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [17271.247534] Lustre: DEBUG MARKER: == sanity test 127d: OSC RPC latency histograms for read and write latency ========================================================== 23:49:55 (1785469795) [17288.202482] Lustre: DEBUG MARKER: == sanity test 127e: client IO latency histograms by size ========================================================== 23:50:12 (1785469812) [17311.362421] Lustre: DEBUG MARKER: == sanity test 127f: OST IO latency histograms by size === 23:50:36 (1785469836) [17350.370981] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 23:51:14 (1785469874) [17361.515703] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 23:51:27 (1785469887) [17388.805858] Lustre: 6541:0:(osd_handler.c:613:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513263, FID: [0x20000040b:0x904e:0x0]) is approaching max size limit [17390.385706] Lustre: 60465:0:(osd_handler.c:613:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513263, FID: [0x20000040b:0x904e:0x0]) is approaching max size limit [17407.358518] Lustre: 6540:0:(osd_handler.c:609:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513263, FID: [0x20000040b:0x904e:0x0]) has reached max size limit [17455.382576] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 23:53:00 (1785469980) [17466.965598] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 23:53:12 (1785469992) [17477.724329] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 23:53:22 (1785470002) [17489.775207] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 23:53:35 (1785470015) [17492.340305] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [17495.125074] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 23:53:40 (1785470020) [17580.102832] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 23:55:05 (1785470105) [17593.754287] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 23:55:19 (1785470119) [17664.909389] Lustre: 14524:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 6549 > trans_max 3200 [17664.920664] Lustre: 14524:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [17664.928256] Lustre: 14524:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 403/403/0, xattr_set: 604/5892/0 [17664.941153] Lustre: 14524:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/142/0, punch: 0/0/0, quota 0/0/0 [17664.950442] Lustre: 14524:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [17664.974315] Lustre: 14524:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [17664.984243] CPU: 2 PID: 14524 Comm: mdt00_004 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [17664.991206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17665.001336] Call Trace: [17665.003702] ? dump_stack+0xbb/0x10e [17665.010309] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [17665.015317] ? dt_trans_start+0x1c/0x70 [obdclass] [17665.021944] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [17665.028201] ? lod_declare_ref_add+0x1a/0x30 [lod] [17665.035156] ? dt_declare_ref_add+0x6c/0x200 [obdclass] [17665.037754] ? lod_trans_start+0x109/0x4c0 [lod] [17665.043879] ? mdd_declare_links_add+0x60/0x90 [mdd] [17665.045474] ? mdd_trans_start+0x18/0x30 [mdd] [17665.046686] ? mdd_unlink+0x7a4/0x13c0 [mdd] [17665.058632] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [17665.063833] ? mdt_reint_rec+0x139/0x2b0 [mdt] [17665.070619] ? mdt_reint_internal+0x693/0xdc0 [mdt] [17665.073782] ? mdt_reint+0x163/0x190 [mdt] [17665.077135] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [17665.079778] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [17665.084778] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [17665.088639] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17665.091429] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [17665.093831] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [17665.096804] ? kthread+0x1d1/0x200 [17665.098306] ? set_kthread_struct+0x70/0x70 [17665.100809] ? ret_from_fork+0x1f/0x30 [17673.893383] Lustre: DEBUG MARKER: == sanity test 130h: FIEMAP deadlock ===================== 23:56:39 (1785470199) [17690.955414] Lustre: DEBUG MARKER: == sanity test 130i: FIEMAP (DoM file) =================== 23:56:56 (1785470216) [17751.954639] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 23:57:56 (1785470276) [17763.496055] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 23:58:08 (1785470288) [17776.347291] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 23:58:20 (1785470300) [17789.834670] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 23:58:34 (1785470314) [17801.080873] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 23:58:46 (1785470326) [17813.728405] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 23:58:58 (1785470338) [17830.801723] Lustre: DEBUG MARKER: sanity test_133a: @@@@@@ FAIL: The close counter on mds1 is 2, not 1 [17980.867879] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 00:01:45 (1785470505) [17998.884515] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 00:02:04 (1785470524) [18044.112813] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 00:02:48 (1785470568) [18100.258386] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 00:03:45 (1785470625) [18114.894809] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 00:04:00 (1785470640) [18152.122219] Lustre: server umount lustre-MDT0000 complete [18152.427703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [18152.435783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18152.453816] LustreError: 82644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18152.467172] LustreError: 82644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [18158.065045] LustreError: 82648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18158.103594] LustreError: 82648:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [18163.119323] LustreError: 81443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785470690 with bad export cookie 13551857542743687818 [18163.125227] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18163.135685] LustreError: 81443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [18163.185749] LustreError: 6541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18163.190243] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18163.203682] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [18163.211951] LustreError: 6541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18163.269786] Lustre: Skipped 4 previous similar messages [18164.171033] Lustre: server umount lustre-MDT0001 complete [18176.705701] Lustre: server umount lustre-OST0000 complete [18188.036186] Lustre: server umount lustre-OST0001 complete [18218.907202] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [18222.891256] Key type lgssc unregistered [18223.340509] LNet: 106909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18223.344344] LNetError: 106909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18223.365080] LNet: Removed LNI 192.168.202.121@tcp [18225.149135] Key type .llcrypt unregistered [18225.155652] Key type ._llcrypt unregistered [18251.224571] Key type ._llcrypt registered [18251.228398] Key type .llcrypt registered [18251.309664] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [18252.732061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18253.203657] alg: No test for adler32 (adler32-zlib) [18255.247920] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [18256.280342] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [18258.415791] Key type lgssc registered [18261.881629] Lustre: Echo OBD driver; http://www.lustre.org/ [18290.465349] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [18308.774826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18308.859725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18310.872349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18318.672355] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [18321.381722] LustreError: 108987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18321.413356] LustreError: 108987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [18326.497026] LustreError: 108988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18331.628900] LustreError: 108987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18335.439352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18335.714627] LustreError: 108988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18336.175344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18343.353787] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [18348.329479] Lustre: 110289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [18361.671204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18362.589149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18371.100771] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [18381.052549] LustreError: 110796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18381.097117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:26155 to 0x280000403:26241) [18381.129618] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:12237 to 0x280000400:12257) [18386.813075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18387.511881] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [18392.596450] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:12081 to 0x2c0000400:12097) [18392.604839] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:24890 to 0x2c0000403:24929) [18398.770147] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [18410.625568] Lustre: DEBUG MARKER: Using TIMEOUT=20 [18415.591641] Lustre: 112473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [18427.307136] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 00:09:12 (1785470952) [18437.717849] LNet: 112967:0:(debug.c:376:cfs_str2mask()) unknown mask ''. [18437.717849] mask usage: [+|-] ... [18438.336225] LNet: 113010:0:(debug.c:376:cfs_str2mask()) unknown mask ''. [18438.336225] mask usage: [+|-] ... [18438.342898] LNet: 113010:0:(debug.c:376:cfs_str2mask()) Skipped 3 previous similar messages [18438.379092] Lustre: DEBUG MARKER:  [18438.390359] Lustre: DEBUG MARKER:  [18439.382525] LNet: 113110:0:(debug.c:376:cfs_str2mask()) unknown mask ''. [18439.382525] mask usage: [+|-] ... [18439.401350] LNet: 113110:0:(debug.c:376:cfs_str2mask()) Skipped 1 previous similar message [18534.352077] LNet: 114668:0:(debug.c:376:cfs_str2mask()) unknown mask ''. [18534.352077] mask usage: [+|-] ... [18534.364736] LNet: 114668:0:(debug.c:376:cfs_str2mask()) Skipped 1 previous similar message [18535.669802] Lustre: DEBUG MARKER:  [18636.265646] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18636.271686] Lustre: Skipped 2 previous similar messages [18636.272811] LustreError: 108987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18636.310996] LustreError: 108987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [18636.492680] Lustre: server umount lustre-MDT0000 complete [18647.646372] LustreError: 108970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785471175 with bad export cookie 515418939425194139 [18647.658313] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18648.220762] Lustre: server umount lustre-MDT0001 complete [18667.999921] Lustre: 107384:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785471179/real 1785471179] req@ffff8a02be695500 x1872201811932288/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785471195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [18668.040459] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18668.060609] Lustre: Skipped 1 previous similar message [18668.959230] Lustre: 107385:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785471180/real 1785471180] req@ffff8a0286479180 x1872201811932544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785471196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [18670.301918] Lustre: server umount lustre-OST0000 complete [18673.190627] Lustre: 107384:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785471184/real 1785471184] req@ffff8a02a1e80e00 x1872201811932800/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785471200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [18679.398113] Lustre: 107384:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785471190/real 1785471190] req@ffff8a02967cd180 x1872201811933440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785471206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [18679.428572] Lustre: 107384:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [18683.424753] Lustre: server umount lustre-OST0001 complete [18711.886462] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [18716.030988] Key type lgssc unregistered [18716.657504] LNet: 118472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18716.664313] LNetError: 118472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18716.701413] LNet: Removed LNI 192.168.202.121@tcp [18718.571145] Key type .llcrypt unregistered [18718.579073] Key type ._llcrypt unregistered [18741.177661] Key type ._llcrypt registered [18741.180169] Key type .llcrypt registered [18741.363850] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [18743.758350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18743.873228] alg: No test for adler32 (adler32-zlib) [18745.386736] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [18746.102621] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [18747.928310] Key type lgssc registered [18750.292473] Lustre: Echo OBD driver; http://www.lustre.org/ [18771.996971] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [18789.659647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18789.703960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18792.470536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18800.465991] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [18801.635482] LustreError: 120553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18801.682564] LustreError: 120553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [18806.763043] LustreError: 120554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18811.890832] LustreError: 120553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18816.996573] LustreError: 120554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18820.701849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18821.554827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18827.656826] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [18831.442041] Lustre: 121853:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [18845.434887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18846.225142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18847.226817] LustreError: 122364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18847.244358] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:12237 to 0x280000400:12289) [18855.040863] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [18857.510354] LustreError: 122361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18857.543964] LustreError: 122361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18861.648508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:26155 to 0x280000403:26273) [18872.070397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18872.575997] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [18877.972856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:24890 to 0x2c0000403:24961) [18877.980162] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:12081 to 0x2c0000400:12129) [18881.384715] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [18896.466617] Lustre: DEBUG MARKER: Using TIMEOUT=20 [18901.883721] Lustre: 124040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [18916.895244] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 00:17:21 (1785471441) [20694.353193] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 00:46:59 (1785473219) [20725.760733] Lustre: *** cfs_fail_loc=327, val=500*** [20727.154273] Lustre: *** cfs_fail_loc=327, val=500*** [20727.160963] Lustre: Skipped 512 previous similar messages [20766.500120] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 00:48:11 (1785473291) [20775.350988] Lustre: *** cfs_fail_loc=328, val=500*** [20777.370279] Lustre: *** cfs_fail_loc=328, val=500*** [20777.372234] Lustre: Skipped 83 previous similar messages [20781.388180] Lustre: *** cfs_fail_loc=328, val=500*** [20781.389804] Lustre: Skipped 210 previous similar messages [20782.492610] Lustre: 120548:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8a028a3ebb80 x1872202304779008/t0(0) o101->33b7dc9e-a703-4cc3-9b1c-8fb78043b64c@192.168.202.21@tcp:245/0 lens 648/0 e 0 to 0 dl 1785473320 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [20783.606201] Lustre: 120549:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8a02a1e80380 x1872202304780160/t0(0) o101->33b7dc9e-a703-4cc3-9b1c-8fb78043b64c@192.168.202.21@tcp:247/0 lens 648/0 e 0 to 0 dl 1785473322 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [20784.743152] Lustre: 120550:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8a02bc638a80 x1872202304780928/t0(0) o101->33b7dc9e-a703-4cc3-9b1c-8fb78043b64c@192.168.202.21@tcp:248/0 lens 648/0 e 0 to 0 dl 1785473323 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [20787.205870] Lustre: 120550:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8a0288f7ad80 x1872202304815488/t0(0) o101->33b7dc9e-a703-4cc3-9b1c-8fb78043b64c@192.168.202.21@tcp:250/0 lens 648/0 e 0 to 0 dl 1785473325 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [20789.412043] Lustre: *** cfs_fail_loc=328, val=500*** [20789.418435] Lustre: Skipped 142 previous similar messages [20792.112874] Lustre: 123746:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8a02982ca680 x1872202304834688/t0(0) o101->33b7dc9e-a703-4cc3-9b1c-8fb78043b64c@192.168.202.21@tcp:255/0 lens 648/0 e 0 to 0 dl 1785473330 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [20792.132676] Lustre: 123746:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) Skipped 3 previous similar messages [20796.255524] LustreError: 187759:0:(ldlm_lockd.c:3246:lock_reclaim_threshold_mb_store()) Failed to set lock_reclaim_threshold_mb, rc = -22. [20821.639856] Lustre: DEBUG MARKER: == sanity test 134c: Lock memory accounting includes associated structures ========================================================== 00:49:06 (1785473346) [20848.561836] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [20850.944323] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [20854.514821] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 00:49:38 (1785473378) [20918.627937] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 00:50:43 (1785473443) [20959.447859] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 00:51:24 (1785473484) [20997.903264] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 00:52:03 (1785473523) [21060.515660] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 00:53:05 (1785473585) [21088.506784] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 00:53:33 (1785473613) [21114.998769] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 00:54:00 (1785473640) [21153.560879] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 00:54:37 (1785473677) [21183.221877] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 00:55:08 (1785473708) [21227.033447] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 00:55:51 (1785473751) [21238.169722] Lustre: DEBUG MARKER: == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 00:56:04 (1785473764) [21264.297647] Lustre: DEBUG MARKER: == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 00:56:29 (1785473789) [21296.202768] Lustre: DEBUG MARKER: == sanity test 150ic: Verify fallocate LARGE zero PREALLOC functionality ========================================================== 00:57:01 (1785473821) [21299.058517] Lustre: DEBUG MARKER: SKIP: sanity test_150ic only check on DoM component [21302.590976] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 00:57:07 (1785473827) [21334.453824] bash (197151): drop_caches: 1 [21353.885451] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 00:57:59 (1785473879) [21363.711730] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 00:58:09 (1785473889) [21371.281367] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 00:58:17 (1785473897) [21385.982476] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 00:58:29 (1785473909) [21402.681843] Lustre: DEBUG MARKER: == sanity test 154C: lfs fid2path on OST FID ============= 00:58:47 (1785473927) [21428.739852] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 00:59:14 (1785473954) [21431.499855] LustreError: 120560:0:(fld_handler.c:268:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [21441.249798] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 00:59:27 (1785473967) [21443.448338] LustreError: 120559:0:(fld_handler.c:268:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [21443.455217] LustreError: 120559:0:(fld_handler.c:268:fld_server_lookup()) Skipped 3 previous similar messages [21453.411484] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 00:59:39 (1785473979) [21462.232745] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 00:59:48 (1785473988) [21477.265634] Lustre: DEBUG MARKER: == sanity test 154db: fid is stored in dir entries ======= 01:00:02 (1785474002) [21496.631349] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 01:00:22 (1785474022) [21507.756168] Lustre: DEBUG MARKER: == sanity test 154ea: .lustre is not returned by readdir (2) ========================================================== 01:00:33 (1785474033) [21589.755917] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 01:01:54 (1785474114) [21602.738302] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 01:02:08 (1785474128) [23607.965538] Lustre: DEBUG MARKER: == sanity test 154h: Verify interactive path2fid ========= 01:35:33 (1785476133) [23619.219411] Lustre: DEBUG MARKER: == sanity test 154i: fid2path for path longer than PATH_MAX ========================================================== 01:35:44 (1785476144) [23687.698722] Lustre: DEBUG MARKER: == sanity test 154j: fid2path for long path crossing MDT boundary ========================================================== 01:36:53 (1785476213) [23719.030801] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 01:37:24 (1785476244) [23737.811761] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 01:37:42 (1785476262) [23761.436469] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 01:38:06 (1785476286) [23781.619154] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 01:38:27 (1785476307) [23799.770627] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 01:38:45 (1785476325) [23858.901175] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 01:39:44 (1785476384) [23926.790176] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 01:40:51 (1785476451) [23989.007738] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 01:41:54 (1785476514) [24061.570841] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 01:43:07 (1785476587) [24077.598776] Lustre: DEBUG MARKER: Turn on read and write cache [24083.639694] Lustre: DEBUG MARKER: Write data and read it back. [24085.799196] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [24091.601462] Lustre: DEBUG MARKER: cache hits: before: 28718, after: 28721 [24093.540478] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [24096.910289] Lustre: DEBUG MARKER: cache hits:: before: 28721, after: 28724 [24099.257766] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [24103.836805] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [24110.360460] Lustre: DEBUG MARKER: cache hits:: before: 28724, after: 28727 [24113.824290] Lustre: DEBUG MARKER: Write data and read it back. [24116.155535] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [24121.983777] Lustre: DEBUG MARKER: cache hits:: before: 28727, after: 28730 [24124.212296] Lustre: DEBUG MARKER: Turn off read and write cache [24131.989314] Lustre: DEBUG MARKER: Write data and read it back [24136.162821] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [24140.766064] Lustre: DEBUG MARKER: cache hits:: before: 28730, after: 28730 [24143.222390] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [24148.124216] Lustre: DEBUG MARKER: Write data and read it back [24150.424657] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [24155.893674] Lustre: DEBUG MARKER: cache hits:: before: 28730, after: 28730 [24157.960575] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [24163.186379] Lustre: DEBUG MARKER: cache hits:: before: 28730, after: 28733 [24175.438440] Lustre: DEBUG MARKER: == sanity test 157a: llapi pool pinning API tests ======== 01:45:01 (1785476701) [24189.386969] Lustre: DEBUG MARKER: == sanity test 157b: lustre.pin inheritance on create ==== 01:45:14 (1785476714) [24204.133314] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 01:45:29 (1785476729) [24207.877525] Lustre: lustre-MDD0000: changelog on [24210.846366] Lustre: lustre-MDD0001: changelog on [24233.245353] Lustre: Failing over lustre-MDT0000 [24233.959746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [24233.968442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [24233.981512] LustreError: 122372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [24233.994720] Lustre: server umount lustre-MDT0000 complete [24233.998018] LustreError: 122372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [24237.037059] LustreError: 122373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [24237.061934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [24242.162978] LustreError: 196445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [24242.198703] LustreError: 196445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [24249.251856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [24249.574451] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [24250.261405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [24250.323171] Lustre: lustre-MDD0000: changelog on [24250.351532] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [24252.493382] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [24255.488294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [24256.433110] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [24256.435046] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [24256.467733] Lustre: Skipped 2 previous similar messages [24256.517965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:27165 to 0x280000403:27201) [24256.520250] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:25851 to 0x2c0000403:25889) [24256.974885] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [24262.892781] Lustre: lustre-MDD0000: changelog off [24265.370451] Lustre: lustre-MDD0001: changelog off [24282.216542] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 01:46:48 (1785476808) [24286.160336] Lustre: lustre-MDD0000: changelog on [24304.367691] Lustre: lustre-MDD0001: changelog off [24308.098759] Lustre: lustre-MDD0000: changelog off [24311.399478] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 01:47:16 (1785476836) [24316.112438] Lustre: lustre-MDD0000: changelog on [24316.119524] Lustre: Skipped 1 previous similar message [24336.252522] Lustre: lustre-MDD0001: changelog off [24343.476801] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 01:47:49 (1785476869) [24348.986782] Lustre: lustre-MDD0000: changelog on [24348.991215] Lustre: Skipped 1 previous similar message [24364.952429] Lustre: lustre-MDD0001: changelog off [24364.970358] Lustre: Skipped 1 previous similar message [24374.597943] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 01:48:19 (1785476899) [24379.411967] Lustre: lustre-MDD0000: changelog on [24379.414664] Lustre: Skipped 1 previous similar message [24396.579441] Lustre: lustre-MDD0001: changelog off [24396.591150] Lustre: Skipped 1 previous similar message [24406.795444] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 01:48:51 (1785476931) [24411.574884] Lustre: lustre-MDD0000: changelog on [24411.579919] Lustre: Skipped 1 previous similar message [24424.124205] Lustre: DEBUG MARKER: 1785476950: creating first dirs [24454.676596] Lustre: *** cfs_fail_loc=1313, val=3*** [24454.681182] Lustre: 120550:0:(mdd_dir.c:995:mdd_changelog_emrg_cleanup()) lustre-MDD0000: changelog has only 3 free catalog entries [24454.687776] Lustre: 120550:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [24454.702157] Lustre: 217201:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 37s with 4 unprocessed records [24494.903430] Lustre: lustre-MDD0001: changelog off [24494.906857] Lustre: Skipped 1 previous similar message [24503.496212] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 01:50:29 (1785477029) [24508.072557] Lustre: lustre-MDD0000: changelog on [24508.080487] Lustre: Skipped 1 previous similar message [24536.730475] Lustre: 200686:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [24536.742176] Lustre: 200686:0:(mdd_dir.c:1082:mdd_changelog_store()) Skipped 1 previous similar message [24536.806062] Lustre: 219668:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 22s with 4 unprocessed records [24536.830047] Lustre: 219668:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [24564.872572] Lustre: lustre-MDD0001: changelog off [24564.891335] Lustre: Skipped 1 previous similar message [24573.498853] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 01:51:38 (1785477098) [24624.039786] Lustre: *** cfs_fail_loc=1316, val=0*** [24624.041253] Lustre: 200686:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [24624.054080] Lustre: 200686:0:(mdd_dir.c:1082:mdd_changelog_store()) Skipped 1 previous similar message [24624.071126] Lustre: 222128:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0001: force deregister of changelog user cl11 idle for 33s with 3 unprocessed records [24624.088799] Lustre: 222128:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) Skipped 2 previous similar messages [24628.287282] Lustre: Failing over lustre-MDT0001 [24628.608968] Lustre: Failing over lustre-MDT0000 [24629.236663] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [24629.252583] Lustre: Skipped 2 previous similar messages [24629.271672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [24631.479402] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping) [24631.502763] Lustre: Skipped 3 previous similar messages [24634.339990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [24634.354665] Lustre: Skipped 4 previous similar messages [24636.563542] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [24640.238845] Lustre: server umount lustre-MDT0001 complete [24640.451211] Lustre: server umount lustre-MDT0000 complete [24651.652703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [24651.794386] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [24652.336185] LustreError: 122372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [24652.358487] LustreError: 122372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [24652.596845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [24652.686447] Lustre: lustre-MDD0000: changelog on [24652.690256] Lustre: Skipped 3 previous similar messages [24652.704388] Lustre: 223316:0:(mdd_device.c:626:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 26 to index 27, being cleared now [24652.739488] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [24659.738477] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [24662.232081] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [24670.184465] LustreError: 223329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [24670.199833] LustreError: 223329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [24670.217943] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [24671.401025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [24672.004746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [24672.101498] Lustre: 224064:0:(mdd_device.c:626:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 20 to index 21, being cleared now [24672.158136] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [24673.422044] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [24677.366925] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [24677.383738] Lustre: Skipped 1 previous similar message [24678.229985] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [24678.547632] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [24678.634874] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:12296 to 0x280000400:12321) [24678.637040] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:12137 to 0x2c0000400:12161) [24678.945332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:27203 to 0x280000403:27233) [24678.953688] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:25851 to 0x2c0000403:25921) [24710.131348] Lustre: lustre-MDD0001: changelog off [24710.136534] Lustre: Skipped 1 previous similar message [24717.782992] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 01:54:03 (1785477243) [24732.800915] LustreError: 226039:0:(mdd_device.c:2065:mdd_changelog_user_purge()) cfs_race id 1315 sleeping [24736.093212] LustreError: 226165:0:(mdd_device.c:1780:mdd_changelog_user_register()) cfs_fail_race id 1315 waking [24736.117428] LustreError: 226039:0:(mdd_device.c:2065:mdd_changelog_user_purge()) cfs_fail_race id 1315 awake: rc=1706 [24739.009860] LustreError: 226359:0:(mdd_device.c:2065:mdd_changelog_user_purge()) cfs_fail_race id 1315 waking [24740.106833] LustreError: 226416:0:(mdd_device.c:1780:mdd_changelog_user_register()) cfs_fail_race id 1315 waking [24773.616834] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 01:54:59 (1785477299) [24812.170604] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 01:55:37 (1785477337) [24823.002751] LustreError: 226922:0:(mdd_dir.c:1050:mdd_changelog_store()) cfs_fail_timeout id 15d sleeping for 3000ms [24826.039108] LustreError: 226922:0:(mdd_dir.c:1050:mdd_changelog_store()) cfs_fail_timeout id 15d awake [24856.293850] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 01:56:21 (1785477381) [24889.108918] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 01:56:54 (1785477414) [24911.557709] LustreError: 226922:0:(mdd_device.c:398:llog_changelog_cancel_cb()) cfs_race id 15f sleeping [24913.617430] LustreError: 223330:0:(mdd_device.c:398:llog_changelog_cancel_cb()) cfs_fail_race id 15f waking [24913.624663] LustreError: 226922:0:(mdd_device.c:398:llog_changelog_cancel_cb()) cfs_fail_race id 15f awake: rc=2944 [24940.598315] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 01:57:46 (1785477466) [24944.982405] Lustre: lustre-MDD0000: changelog on [24944.990557] Lustre: Skipped 13 previous similar messages [27752.291719] LustreError: 223698:0:(mdd_device.c:412:llog_changelog_cancel_cb()) cfs_race id 16c sleeping [27754.267893] LustreError: 223328:0:(llog_osd.c:1130:llog_osd_next_block()) cfs_fail_race id 16c waking [27754.297929] LustreError: 223698:0:(mdd_device.c:412:llog_changelog_cancel_cb()) cfs_fail_race id 16c awake: rc=3024 [27775.197627] Lustre: lustre-MDD0001: changelog off [27775.204310] Lustre: Skipped 13 previous similar messages [27784.184138] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 02:45:09 (1785480309) [27789.528697] Lustre: lustre-MDD0000: changelog on [27789.530062] Lustre: Skipped 1 previous similar message [27794.861303] LustreError: 234431:0:(mdd_device.c:1734:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [27796.470548] Lustre: 234479:0:(mdd_device.c:1751:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [27798.465042] LustreError: 234527:0:(mdd_device.c:1743:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [27843.779121] Lustre: lustre-MDD0000: changelog off [27843.783485] Lustre: Skipped 1 previous similar message [27857.680954] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 02:46:22 (1785480382) [27863.679906] Lustre: lustre-MDD0000: changelog on [27863.687380] Lustre: Skipped 1 previous similar message [27874.382391] Lustre: Failing over lustre-MDT0000 [27875.021335] Lustre: server umount lustre-MDT0000 complete [27875.809773] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [27875.831239] LustreError: 226922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [27875.846595] Lustre: Skipped 6 previous similar messages [27875.903859] LustreError: 226922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [27880.946023] LustreError: 226921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [27880.958956] LustreError: 226921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [27891.036172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [27891.176438] LustreError: 226922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [27891.206799] LustreError: 226922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [27891.514669] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [27891.870074] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (not set up) [27891.872634] Lustre: Skipped 5 previous similar messages [27892.391049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [27892.753463] Lustre: 236866:0:(mdd_device.c:626:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 90150 to index 18446744073709551615, being cleared now [27892.822540] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [27893.113660] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [27897.854467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [27897.860415] Lustre: Skipped 3 previous similar messages [27897.913882] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [27897.927495] Lustre: Skipped 1 previous similar message [27898.003125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:27235 to 0x280000403:27265) [27898.003779] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:25923 to 0x2c0000403:25953) [27900.169059] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [27915.473995] Lustre: lustre-MDD0001: changelog off [27915.475850] Lustre: Skipped 1 previous similar message [27920.870648] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 02:47:26 (1785480446) [27941.132477] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records * time ========================================================== 02:47:46 (1785480466) [27968.296776] Lustre: 233594:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [27968.312944] Lustre: 233594:0:(mdd_dir.c:1082:mdd_changelog_store()) Skipped 1 previous similar message [27968.361410] Lustre: 238866:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl2 idle for 864020s with 500000004 unprocessed records [28002.702246] Lustre: DEBUG MARKER: == sanity test 160t: changelog garbage collect on lack of space ========================================================== 02:48:47 (1785480527) [28007.283472] Lustre: lustre-MDD0000: changelog on [28007.289783] Lustre: Skipped 4 previous similar messages [28100.072199] Lustre: *** cfs_fail_loc=18c, val=1210732*** [28100.110807] Lustre: Skipped 1 previous similar message [28100.132247] Lustre: 223328:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [28100.150435] Lustre: 223328:0:(mdd_dir.c:1082:mdd_changelog_store()) Skipped 1 previous similar message [28100.199938] Lustre: 240895:0:(mdd_dir.c:966:mdd_changelog_is_space_safe()) lustre-MDD0000: changelog size 1MB with 1MB space limit [28100.246810] Lustre: 240895:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl3-user1 idle for 93s with 7506 unprocessed records [28100.282818] Lustre: 240895:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [28134.503321] Lustre: lustre-MDD0000: changelog off [28134.507146] Lustre: Skipped 3 previous similar messages [28195.869473] Lustre: DEBUG MARKER: == sanity test 160u: changelog rename record type name and sname strings are correct ========================================================== 02:52:00 (1785480720) [28228.381280] Lustre: DEBUG MARKER: == sanity test 160v: setattr preserves mtime update despite inter-client clock skew ========================================================== 02:52:33 (1785480753) [28231.164538] Lustre: DEBUG MARKER: SKIP: sanity test_160v need >= 2 clients [28234.509107] Lustre: DEBUG MARKER: == sanity test 160w: lfs changelog --user --mask ========= 02:52:39 (1785480759) [28288.654697] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 02:53:33 (1785480813) [28346.865408] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 02:54:32 (1785480872) [28405.531297] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 02:55:28 (1785480928) [28413.605194] Lustre: lustre-MDD0000: changelog on [28413.617834] Lustre: Skipped 5 previous similar messages [28437.432545] Lustre: lustre-MDD0001: changelog off [28437.435428] Lustre: Skipped 5 previous similar messages [28443.918654] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 02:56:09 (1785480969) [28472.370141] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 02:56:38 (1785480998) [28483.091064] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 02:56:48 (1785481008) [28496.632989] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 02:57:02 (1785481022) [28708.577959] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 03:00:33 (1785481233) [28719.367336] Lustre: Failing over lustre-OST0000 [28720.030949] Lustre: server umount lustre-OST0000 complete [28720.345439] LustreError: 200005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [28722.174658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [28725.392263] LustreError: 187048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [28725.450616] LustreError: 187048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [28730.523850] LustreError: 122361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [28730.568290] LustreError: 122361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [28740.752981] LustreError: 187050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [28740.783984] LustreError: 187050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [28751.286528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [28752.132676] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [28752.165264] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [28753.355433] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [28753.729441] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [28753.730355] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [28753.746919] Lustre: Skipped 3 previous similar messages [28766.718389] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [28773.139127] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 03:01:38 (1785481298) [28812.036812] Lustre: Failing over lustre-OST0000 [28812.167894] Lustre: server umount lustre-OST0000 complete [28812.498334] LustreError: 187050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [28812.536288] LustreError: 187050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [28813.810545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [28813.823430] Lustre: Skipped 1 previous similar message [28824.383700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [28825.230704] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [28825.273445] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [28826.673155] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [28826.787159] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [28826.787868] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [28826.799138] Lustre: Skipped 1 previous similar message [28840.665314] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [28846.870557] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 03:02:51 (1785481371) [28889.166612] Lustre: Failing over lustre-OST0000 [28889.277422] LustreError: 200009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [28889.289508] LustreError: 200009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [28889.302160] Lustre: server umount lustre-OST0000 complete [28892.170777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [28892.192426] Lustre: Skipped 1 previous similar message [28912.680053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [28913.305968] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [28913.329630] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [28914.944404] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [28915.144494] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [28915.145198] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [28915.173216] Lustre: Skipped 1 previous similar message [28922.365549] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [28927.180239] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 03:04:12 (1785481452) [28988.184140] Lustre: Failing over lustre-OST0000 [28988.481680] Lustre: server umount lustre-OST0000 complete [28989.640143] LustreError: 200005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [28989.671658] LustreError: 200005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [28990.453292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [28990.477567] Lustre: Skipped 2 previous similar messages [28998.833233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [28999.616717] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [28999.635713] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [28999.830238] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [29001.502341] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [29001.504554] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [29001.508574] Lustre: Skipped 1 previous similar message [29009.710306] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [29017.332784] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 03:05:41 (1785481541) [29047.212036] Lustre: Failing over lustre-OST0000 [29047.517737] Lustre: server umount lustre-OST0000 complete [29047.779553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [29047.793217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [29060.684652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [29061.111098] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [29061.394909] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [29061.427502] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [29062.731893] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [29063.265987] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [29063.268855] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [29063.307249] Lustre: Skipped 1 previous similar message [29071.801872] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [29078.702620] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 03:06:43 (1785481603) [29087.261540] Lustre: Failing over lustre-OST0000 [29087.275399] LustreError: 256822:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000403:0x91a4:0x0].0x0 (ffff8a0385bacb00) refcount nonzero (2) after lock cleanup; forcing cleanup. [29089.270435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [29089.289678] LustreError: Skipped 1 previous similar message [29089.301468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [29089.336438] Lustre: Skipped 1 previous similar message [29089.370434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [29089.395163] Lustre: Skipped 1 previous similar message [29089.524717] Lustre: server umount lustre-OST0000 complete [29114.897988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [29115.541517] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [29115.572674] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [29117.235241] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [29117.719050] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [29117.720431] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [29117.727219] Lustre: Skipped 1 previous similar message [29132.743111] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [29146.026728] Lustre: DEBUG MARKER: == sanity test 165g: ofd_access_log_reader --keepalive works ========================================================== 03:07:48 (1785481668) [29194.317681] Lustre: Failing over lustre-OST0000 [29194.435274] Lustre: lustre-OST0000: Not available for connect from 192.168.202.21@tcp (stopping) [29195.233131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [29195.250579] LustreError: Skipped 1 previous similar message [29195.260134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [29195.288465] Lustre: Skipped 1 previous similar message [29196.867840] Lustre: server umount lustre-OST0000 complete [29199.503276] LustreError: 200008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [29199.536925] LustreError: 200008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 28 previous similar messages [29223.884976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [29224.144037] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [29224.175140] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [29225.108343] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [29226.138681] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [29226.142167] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [29226.146342] Lustre: Skipped 1 previous similar message [29233.335817] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug all all [29240.170873] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 03:09:24 (1785481764) [29243.221323] Lustre: DEBUG MARKER: creating a 10 Mb file [29340.137727] Lustre: DEBUG MARKER: starting reads [29342.913070] Lustre: DEBUG MARKER: truncating the file [29346.395436] Lustre: DEBUG MARKER: killing dd [29349.902382] Lustre: DEBUG MARKER: removing the temporary file [29366.320422] Lustre: DEBUG MARKER: == sanity test 170a: test lctl df to handle corrupted log ========================================================== 03:11:30 (1785481890) [29383.280734] Lustre: DEBUG MARKER: == sanity test 170b: check filename encoding ============= 03:11:47 (1785481907) [29436.810155] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 03:12:41 (1785481961) [29453.551682] Lustre: DEBUG MARKER: == sanity test 172: manual device removal with lctl cleanup/detach ================================================================ 03:12:59 (1785481979) [29466.559587] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 03:13:12 (1785481992) [29468.540285] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [29470.969272] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 03:13:16 (1785481996) [29475.994877] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_module obdecho/obdecho [29508.495778] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 03:13:53 (1785482033) [29515.973134] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_module obdecho/obdecho [29516.384855] Lustre: Echo OBD driver; http://www.lustre.org/ [29551.970586] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 03:14:37 (1785482077) [29670.285088] Lustre: DEBUG MARKER: == sanity test 182a: Test parallel modify metadata operations from mdc ========================================================== 03:16:35 (1785482195) [29830.011389] Lustre: DEBUG MARKER: == sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 03:19:15 (1785482355) [30917.069269] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 03:37:23 (1785483443) [30918.904528] Lustre: *** cfs_fail_loc=148, val=0*** [30931.787358] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 03:37:36 (1785483456) [30954.247990] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 03:37:59 (1785483479) [30968.839929] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 03:38:13 (1785483493) [31060.303641] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 03:39:45 (1785483585) [31075.070831] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 03:40:00 (1785483600) [31090.014893] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 03:40:15 (1785483615) [31099.384698] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 03:40:25 (1785483625) [31110.601460] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 03:40:36 (1785483636) [31131.728693] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 03:40:55 (1785483655) [31150.952984] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 03:41:16 (1785483676) [31162.325963] Lustre: DEBUG MARKER: == sanity test 190a: check lfs project -p works with project name ========================================================== 03:41:27 (1785483687) [31172.611985] Lustre: DEBUG MARKER: == sanity test 190b: check lfs find --project works with project name ========================================================== 03:41:37 (1785483697) [31224.102267] Lustre: DEBUG MARKER: == sanity test 190c: check lfs project -p works with u:USERNAME ========================================================== 03:42:29 (1785483749) [31247.404738] Lustre: DEBUG MARKER: == sanity test complete, duration 30840 sec ============== 03:42:52 (1785483772) [31250.129868] Lustre: DEBUG MARKER: === sanity: start cleanup 03:42:55 (1785483775) === [31328.236353] Lustre: DEBUG MARKER: === sanity: finish cleanup 03:44:11 (1785483851) === [31335.907930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [31335.922723] Lustre: Skipped 1 previous similar message [31335.954206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [31335.968278] Lustre: Skipped 5 previous similar messages [31341.027319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [31343.245690] Lustre: server umount lustre-MDT0000 complete [31346.154577] LustreError: 267061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [31346.206403] LustreError: 267061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [31360.119449] LustreError: 186832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785483887 with bad export cookie 7861045795145040517 [31360.122428] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [31360.131530] LustreError: 186832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [31361.506562] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [31361.537768] Lustre: Skipped 3 previous similar messages [31361.881212] Lustre: server umount lustre-MDT0001 complete [31380.950474] Lustre: server umount lustre-OST0000 complete [31393.574495] Lustre: server umount lustre-OST0001 complete [31417.597861] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [31421.942213] Key type lgssc unregistered [31422.371202] LNet: 279778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [31422.384570] LNetError: 279778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [31422.403293] LNet: Removed LNI 192.168.202.121@tcp [31423.732241] Key type .llcrypt unregistered [31423.738121] Key type ._llcrypt unregistered