[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487032791 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002362] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009122] LSM: Security Framework initializing [ 0.010048] Yama: becoming mindful. [ 0.011032] SELinux: Initializing. [ 0.012060] *** VALIDATE selinux *** [ 0.019232] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023835] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025132] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027115] *** VALIDATE tmpfs *** [ 0.029427] *** VALIDATE proc *** [ 0.030219] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.032250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035026] Spectre V2 : User space: Vulnerable [ 0.036009] Speculative Store Bypass: Vulnerable [ 0.039535] debug: unmapping init [mem 0xffffffffb4a59000-0xffffffffb4a60fff] [ 0.041162] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042670] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043022] ... version: 2 [ 0.044011] ... bit width: 48 [ 0.045010] ... generic registers: 4 [ 0.046012] ... value mask: 0000ffffffffffff [ 0.047011] ... max period: 00007fffffffffff [ 0.048013] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.050273] rcu: Hierarchical SRCU implementation. [ 0.052389] smp: Bringing up secondary CPUs ... [ 0.053521] x86: Booting SMP configuration: [ 0.054021] .... node #0, CPUs: #1 #2 #3 [ 0.057789] smp: Brought up 1 node, 4 CPUs [ 0.059012] smpboot: Max logical packages: 1 [ 0.060013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143574] node 0 deferred pages initialised in 81ms [ 0.146111] devtmpfs: initialized [ 0.147214] x86/mm: Memory block size: 128MB [ 0.150773] gcov: version magic: 0x41383552 [ 0.152399] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157280] pinctrl core: initialized pinctrl subsystem [ 0.159339] [ 0.159766] ************************************************************* [ 0.162012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164009] ** ** [ 0.166010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168009] ** ** [ 0.170009] ** This means that this kernel is built to expose internal ** [ 0.172010] ** IOMMU data structures, which may compromise security on ** [ 0.174009] ** your system. ** [ 0.175008] ** ** [ 0.177009] ** If you see this message and you are not debugging the ** [ 0.179008] ** kernel, report this immediately to your vendor! ** [ 0.181009] ** ** [ 0.182008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184010] ************************************************************* [ 0.187586] NET: Registered protocol family 16 [ 0.188468] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196085] cpuidle: using governor menu [ 0.198013] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.200537] PCI: Using configuration type 1 for base access [ 0.202119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.211048] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216119] cryptd: max_cpu_qlen set to 1000 [ 0.220257] ACPI: Added _OSI(Module Device) [ 0.222016] ACPI: Added _OSI(Processor Device) [ 0.224012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233190] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240414] ACPI: Interpreter enabled [ 0.242048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243008] ACPI: Using IOAPIC for interrupt routing [ 0.244095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247285] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.259114] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.262038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.265027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273384] acpiphp: Slot [2] registered [ 0.275105] acpiphp: Slot [5] registered [ 0.277066] acpiphp: Slot [6] registered [ 0.279097] acpiphp: Slot [7] registered [ 0.280139] acpiphp: Slot [8] registered [ 0.282095] acpiphp: Slot [9] registered [ 0.284109] acpiphp: Slot [10] registered [ 0.285108] acpiphp: Slot [3] registered [ 0.287125] acpiphp: Slot [4] registered [ 0.288070] acpiphp: Slot [11] registered [ 0.289075] acpiphp: Slot [12] registered [ 0.291081] acpiphp: Slot [13] registered [ 0.292077] acpiphp: Slot [14] registered [ 0.293093] acpiphp: Slot [15] registered [ 0.294069] acpiphp: Slot [16] registered [ 0.295082] acpiphp: Slot [17] registered [ 0.296069] acpiphp: Slot [18] registered [ 0.298080] acpiphp: Slot [19] registered [ 0.299140] acpiphp: Slot [20] registered [ 0.301108] acpiphp: Slot [21] registered [ 0.303094] acpiphp: Slot [22] registered [ 0.304099] acpiphp: Slot [23] registered [ 0.306093] acpiphp: Slot [24] registered [ 0.308101] acpiphp: Slot [25] registered [ 0.310091] acpiphp: Slot [26] registered [ 0.311121] acpiphp: Slot [27] registered [ 0.313091] acpiphp: Slot [28] registered [ 0.315093] acpiphp: Slot [29] registered [ 0.316142] acpiphp: Slot [30] registered [ 0.318103] acpiphp: Slot [31] registered [ 0.320055] PCI host bridge to bus 0000:00 [ 0.322019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.325021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.331019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.337024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338156] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.340934] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.344197] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352783] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.358009] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.361017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370557] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373723] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.377044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.380748] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.406012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.412538] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.425015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.435013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.459021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470833] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.482035] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.493016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.514016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.528449] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.536025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.546022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.562018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.572193] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.581015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.593015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.618015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.631950] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.641021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.647023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.663016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.672349] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.681014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.696014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.716014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.728827] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.731579] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.735410] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.738328] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.740391] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.744119] iommu: Default domain type: Passthrough [ 0.746412] SCSI subsystem initialized [ 0.748156] ACPI: bus type USB registered [ 0.750110] usbcore: registered new interface driver usbfs [ 0.752084] usbcore: registered new interface driver hub [ 0.754077] usbcore: registered new device driver usb [ 0.756179] pps_core: LinuxPPS API ver. 1 registered [ 0.759013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.762054] PTP clock support registered [ 0.765037] EDAC MC: Ver: 3.0.0 [ 0.767176] PCI: Using ACPI for IRQ routing [ 0.768814] NetLabel: Initializing [ 0.769009] NetLabel: domain hash size = 128 [ 0.770010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.771086] NetLabel: unlabeled traffic allowed by default [ 0.773111] vgaarb: loaded [ 0.775345] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.777013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.782749] clocksource: Switched to clocksource kvm-clock [ 0.901390] VFS: Disk quotas dquot_6.6.0 [ 0.902973] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.905463] *** VALIDATE ramfs *** [ 0.906724] *** VALIDATE hugetlbfs *** [ 0.908286] pnp: PnP ACPI init [ 0.911382] pnp: PnP ACPI: found 6 devices [ 0.927130] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.930542] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.932784] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.935088] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.937493] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.939773] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.942881] NET: Registered protocol family 2 [ 0.945373] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.952733] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.957489] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.964094] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.968415] TCP: Hash tables configured (established 65536 bind 65536) [ 0.971255] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.975288] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.978750] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.982504] NET: Registered protocol family 1 [ 0.985139] RPC: Registered named UNIX socket transport module. [ 0.987298] RPC: Registered udp transport module. [ 0.989158] RPC: Registered tcp transport module. [ 0.991073] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.993676] NET: Registered protocol family 44 [ 0.995502] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.997775] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.000034] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.002501] PCI: CLS 0 bytes, default 64 [ 1.004211] Unpacking initramfs... [ 2.408876] debug: unmapping init [mem 0xffff9e77bcc54000-0xffff9e77bffbffff] [ 2.413053] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.415611] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.418908] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.919537] Initialise system trusted keyrings [ 2.921441] Key type blacklist registered [ 2.924257] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.935925] zbud: loaded [ 2.939958] *** VALIDATE nfs *** [ 2.941148] *** VALIDATE nfs4 *** [ 2.942611] pstore: using deflate compression [ 2.946300] Platform Keyring initialized [ 3.058257] NET: Registered protocol family 38 [ 3.060393] Key type asymmetric registered [ 3.062246] Asymmetric key parser 'x509' registered [ 3.064205] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.067963] io scheduler mq-deadline registered [ 3.072529] io scheduler kyber registered [ 3.074188] io scheduler bfq registered [ 3.076340] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.079027] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.081508] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.084070] ACPI: Power Button [PWRF] [ 3.089102] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.095476] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.111276] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.122774] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.137372] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.166902] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.197742] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.203924] Non-volatile memory driver v1.3 [ 3.206711] Linux agpgart interface v0.103 [ 3.242019] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.245219] vda: detected capacity change from 0 to 74706944 [ 3.260129] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.263545] vdb: detected capacity change from 0 to 1073741824 [ 3.278419] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.282157] vdc: detected capacity change from 0 to 2621440000 [ 3.298433] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.301485] vdd: detected capacity change from 0 to 2621440000 [ 3.320600] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.323969] vde: detected capacity change from 0 to 4294967296 [ 3.340521] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.344117] vdf: detected capacity change from 0 to 4294967296 [ 3.355326] libphy: Fixed MDIO Bus: probed [ 3.366342] usbcore: registered new interface driver usbserial_generic [ 3.368712] usbserial: USB Serial support registered for generic [ 3.371254] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.376175] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.378171] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.380222] mousedev: PS/2 mouse device common for all mice [ 3.383243] rtc_cmos 00:05: RTC can wake from S4 [ 3.388865] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.389260] rtc_cmos 00:05: registered as rtc0 [ 3.394925] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.398117] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.400953] intel_pstate: CPU model not supported [ 3.407686] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.412729] hid: raw HID events driver (C) Jiri Kosina [ 3.414762] usbcore: registered new interface driver usbhid [ 3.416975] usbhid: USB HID core driver [ 3.418748] drop_monitor: Initializing network drop monitor service [ 3.421270] Initializing XFRM netlink socket [ 3.423408] NET: Registered protocol family 10 [ 3.426491] Segment Routing with IPv6 [ 3.428161] NET: Registered protocol family 17 [ 3.430711] mpls_gso: MPLS GSO support [ 3.436929] RAS: Correctable Errors collector initialized. [ 3.439172] AVX version of gcm_enc/dec engaged. [ 3.440837] AES CTR mode by8 optimization enabled [ 3.506797] sched_clock: Marking stable (3506673192, 0)->(4435661219, -928988027) [ 3.510144] registered taskstats version 1 [ 3.511920] Loading compiled-in X.509 certificates [ 3.514033] zswap: loaded using pool lzo/zbud [ 3.541521] Key type big_key registered [ 3.553518] Key type encrypted registered [ 3.555852] ima: No TPM chip found, activating TPM-bypass! [ 3.559398] ima: Allocated hash algorithm: sha1 [ 3.561800] ima: No architecture policies found [ 3.564151] evm: Initialising EVM extended attributes: [ 3.566293] evm: security.selinux [ 3.567606] evm: security.ima [ 3.568804] evm: security.capability [ 3.570185] evm: HMAC attrs: 0x1 [ 3.573686] rtc_cmos 00:05: setting system clock to 2026-08-11 03:12:13 UTC (1786417933) [ 3.579600] debug: unmapping init [mem 0xffffffffb5a03000-0xffffffffb5bfffff] [ 3.583059] debug: unmapping init [mem 0xffffffffb4782000-0xffffffffb4a58fff] [ 3.587148] Write protecting the kernel read-only data: 28672k [ 3.592095] debug: unmapping init [mem 0xffffffffb2e03000-0xffffffffb2ffffff] [ 3.594980] debug: unmapping init [mem 0xffffffffb3714000-0xffffffffb37fffff] [ 3.627271] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.637183] systemd[1]: Detected virtualization kvm. [ 3.639928] systemd[1]: Detected architecture x86-64. [ 3.642237] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.670067] systemd[1]: No hostname configured. [ 3.672122] systemd[1]: Set hostname to . [ 3.673818] random: systemd: uninitialized urandom read (16 bytes read) [ 3.675958] systemd[1]: Initializing machine ID from random generator. [ 3.820146] random: systemd: uninitialized urandom read (16 bytes read) [ 3.823108] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.827689] random: systemd: uninitialized urandom read (16 bytes read) [ 3.830161] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.834213] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.457611] device-mapper: uevent: version 1.0.3 [ 4.461691] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 4.811035] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.201745] scsi host0: ata_piix [ 5.211528] virtio_net virtio0 ens2: renamed from eth0 [ 5.221251] scsi host1: ata_piix [ 5.223982] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.227991] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.541964] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.075589] random: crng init done [ 10.077422] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.471482] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.608939] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.862922] SELinux: Disabled at runtime. [ 11.924990] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.934652] systemd[1]: Detected virtualization kvm. [ 11.936847] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.452976] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.456393] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.460941] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.465986] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.468455] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.476362] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.485764] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK [ 12.518787] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.926878] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.207090] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.255961] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.328596] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.357757] EDAC sbridge: Ver: 1.1.2 [ 15.184773] Key type dns_resolver registered [ 15.489678] NFS: Registering the id_resolver key type [ 15.491285] Key type id_resolver registered [ 15.492480] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (5s / no limit) [** ] A start job is running for Configur…-only root support (6s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ 29.409051] hrtimer: interrupt took 2149599 ns Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg606-server login: [ 74.522154] libcfs: loading out-of-tree module taints kernel. [ 74.587297] Key type ._llcrypt registered [ 74.589017] Key type .llcrypt registered [ 74.703371] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 92.813503] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 94.565831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 94.583593] alg: No test for adler32 (adler32-zlib) [ 96.162451] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 97.078351] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 98.823180] Key type lgssc registered [ 100.817932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.034276] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.273623] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 168.772552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 168.811623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.095427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 170.128908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 170.221337] Lustre: lustre-MDT0000: new disk, initializing [ 170.316251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.347051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 176.925520] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.355443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.531156] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 191.593951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 191.608994] Lustre: Skipped 1 previous similar message [ 191.751948] Lustre: lustre-MDT0001: new disk, initializing [ 191.855251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 191.887370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 191.899748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.817716] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.722565] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 212.866120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.188933] Lustre: lustre-OST0000: new disk, initializing [ 213.196348] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 213.205790] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.265218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.648512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 218.661817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.743067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 219.347505] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.472063] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.620631] Lustre: lustre-OST0001: new disk, initializing [ 233.623641] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 233.628081] Lustre: 9517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.724722] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 239.915995] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.195415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 240.210466] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 240.347225] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 252.051548] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 258.070612] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 264.434943] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing check_logdir /tmp/testlogs/ [ 269.857063] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing yml_node [ 275.017531] Lustre: DEBUG MARKER: Client: 2.17.56.52 [ 278.221751] Lustre: DEBUG MARKER: MDS: 2.17.56.52 [ 280.823478] Lustre: DEBUG MARKER: OSS: 2.17.56.52 [ 282.530327] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-quota ============----- Mon Aug 10 23:16:49 EDT 2026 [ 298.330956] Lustre: DEBUG MARKER: excepting tests: 2 4a 63 65 [ 299.813024] Lustre: DEBUG MARKER: skipping tests SLOW=no: 61 [ 302.792801] Lustre: DEBUG MARKER: === sanity-quota: start setup 23:17:10 (1786418230) === [ 307.801983] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing check_config_client /mnt/lustre [ 325.063388] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 328.723857] Lustre: 13425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 332.682253] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 338.652521] Lustre: DEBUG MARKER: === sanity-quota: finish setup 23:17:45 (1786418265) === [ 400.801854] Lustre: DEBUG MARKER: == sanity-quota test 0: Test basic quota performance ===== 23:18:47 (1786418327) [ 439.444510] Lustre: DEBUG MARKER: == sanity-quota test 1a: Block hard limit (normal use and out of quota) ========================================================== 23:19:26 (1786418366) [ 449.877328] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 456.074760] Lustre: DEBUG MARKER: Write... [ 457.772900] Lustre: DEBUG MARKER: Write out of block quota ... [ 491.245602] Lustre: DEBUG MARKER: -------------------------------------- [ 492.951890] Lustre: DEBUG MARKER: Group quota (block hardlimit:10 MB) [ 499.961376] Lustre: DEBUG MARKER: Write... [ 502.235715] Lustre: DEBUG MARKER: Write out of block quota ... [ 538.162704] Lustre: DEBUG MARKER: -------------------------------------- [ 540.053160] Lustre: DEBUG MARKER: Project quota (block hardlimit:10 mb) [ 542.837498] Lustre: DEBUG MARKER: Write... [ 545.212610] Lustre: DEBUG MARKER: Write out of block quota ... [ 588.950121] Lustre: DEBUG MARKER: == sanity-quota test 1b: Quota pools: Block hard limit (normal use and out of quota) ========================================================== 23:21:56 (1786418516) [ 597.618640] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 614.616318] Lustre: DEBUG MARKER: Write... [ 616.689283] Lustre: DEBUG MARKER: Write out of block quota ... [ 646.743281] Lustre: DEBUG MARKER: -------------------------------------- [ 648.246886] Lustre: DEBUG MARKER: Group quota (block hardlimit:20 MB) [ 653.930989] Lustre: DEBUG MARKER: Write... [ 655.673419] Lustre: DEBUG MARKER: Write out of block quota ... [ 686.406586] Lustre: DEBUG MARKER: -------------------------------------- [ 687.775626] Lustre: DEBUG MARKER: Project quota (block hardlimit:20 mb) [ 689.915458] Lustre: DEBUG MARKER: Write... [ 692.057378] Lustre: DEBUG MARKER: Write out of block quota ... [ 746.697244] Lustre: DEBUG MARKER: == sanity-quota test 1c: Quota pools: check 3 pools with hardlimit only for global ========================================================== 23:24:34 (1786418674) [ 757.829221] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 785.161688] Lustre: DEBUG MARKER: Write... [ 788.025740] Lustre: DEBUG MARKER: Write out of block quota ... [ 873.266811] Lustre: DEBUG MARKER: == sanity-quota test 1d: Quota pools: check block hardlimit on different pools ========================================================== 23:26:40 (1786418800) [ 887.308309] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 915.919113] Lustre: DEBUG MARKER: Write... [ 918.344407] Lustre: DEBUG MARKER: Write out of block quota ... [ 1001.589488] Lustre: DEBUG MARKER: == sanity-quota test 1e: Quota pools: global pool high block limit vs quota pool with small ========================================================== 23:28:48 (1786418928) [ 1017.080022] Lustre: DEBUG MARKER: User quota (block hardlimit:53000000 MB) [ 1032.996963] Lustre: DEBUG MARKER: Write... [ 1035.270229] Lustre: DEBUG MARKER: Write out of block quota ... [ 1047.690833] Lustre: DEBUG MARKER: Write... [ 1093.685760] Lustre: DEBUG MARKER: == sanity-quota test 1f: Quota pools: correct qunit after removing/adding OST ========================================================== 23:30:21 (1786419021) [ 1103.210814] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 1118.069901] Lustre: DEBUG MARKER: Write... [ 1120.138145] Lustre: DEBUG MARKER: Write out of block quota ... [ 1148.504940] Lustre: DEBUG MARKER: Write... [ 1151.304893] Lustre: DEBUG MARKER: Write out of block quota ... [ 1197.423804] Lustre: DEBUG MARKER: == sanity-quota test 1g: Quota pools: Block hard limit with wide striping ========================================================== 23:32:04 (1786419124) [ 1210.354462] Lustre: DEBUG MARKER: User quota (block hardlimit:40 MB) [ 1225.902851] Lustre: 6514:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 6570 > trans_max 3200 [ 1225.909178] Lustre: 6514:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 0/0/0 [ 1225.915140] Lustre: 6514:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 401/401/0, xattr_set: 602/5615/0 [ 1225.921366] Lustre: 6514:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/142/0, punch: 0/0/0, quota 8/328/0 [ 1225.926571] Lustre: 6514:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/68/0, delete: 0/0/0 [ 1225.931828] Lustre: 6514:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1225.938361] CPU: 2 PID: 6514 Comm: mdt00_000 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1225.947683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1225.953206] Call Trace: [ 1225.954567] ? dump_stack+0xbb/0x10e [ 1225.958227] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [ 1225.961267] ? dt_trans_start+0x1c/0x70 [obdclass] [ 1225.964958] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [ 1225.968604] ? lod_xattr_list+0x30/0x30 [lod] [ 1225.971618] ? lod_trans_start+0x109/0x4c0 [lod] [ 1225.976743] ? mdd_declare_attr_set+0x91/0x880 [mdd] [ 1225.981847] ? mdd_env_info+0x25/0xc0 [mdd] [ 1225.983747] ? mdd_trans_start+0x18/0x30 [mdd] [ 1225.989811] ? mdd_attr_set+0xa5a/0x1370 [mdd] [ 1225.991629] ? mdt_reint_setattr+0x14aa/0x2030 [mdt] [ 1225.995370] ? mdt_reint_setattr+0x14aa/0x2030 [mdt] [ 1226.000266] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 1226.003484] ? mdt_reint_internal+0x693/0xdc0 [mdt] [ 1226.008362] ? mdt_reint+0x163/0x190 [mdt] [ 1226.012707] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 1226.018361] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [ 1226.023439] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 1226.030348] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 1226.039212] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 1226.042669] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [ 1226.048241] ? kthread+0x1d1/0x200 [ 1226.051634] ? set_kthread_struct+0x70/0x70 [ 1226.056329] ? ret_from_fork+0x1f/0x30 [ 1227.934735] Lustre: DEBUG MARKER: Write... [ 1241.275406] Lustre: DEBUG MARKER: Write out of block quota ... [ 1316.646286] Lustre: DEBUG MARKER: == sanity-quota test 1h: Block hard limit test using fallocate ========================================================== 23:34:03 (1786419243) [ 1329.126826] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 1336.545324] Lustre: DEBUG MARKER: Write 5MiB Using Fallocate [ 1343.936685] Lustre: DEBUG MARKER: Write 11MiB Using Fallocate [ 1423.202530] Lustre: DEBUG MARKER: == sanity-quota test 1i: Quota pools: different limit and usage relations ========================================================== 23:35:50 (1786419350) [ 1435.118344] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 1449.431769] Lustre: DEBUG MARKER: Write... [ 1451.651671] Lustre: DEBUG MARKER: Write out of block quota ... [ 1483.368483] Lustre: DEBUG MARKER: Write... [ 1485.226791] Lustre: DEBUG MARKER: Write out of block quota ... [ 1495.480882] LustreError: 6516:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:14344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 1536.224355] Lustre: DEBUG MARKER: == sanity-quota test 1j: Enable project quota enforcement for root ========================================================== 23:37:43 (1786419463) [ 1550.555251] Lustre: DEBUG MARKER: -------------------------------------- [ 1552.307094] Lustre: DEBUG MARKER: Project quota (hardlimit: 40 mb, 2048 files) [ 1893.545858] Lustre: DEBUG MARKER: == sanity-quota test 1k: LQA: Block hard limit (normal use and out of quota) ========================================================== 23:43:40 (1786419820) [ 1906.561532] Lustre: DEBUG MARKER: Write... [ 1909.426644] Lustre: DEBUG MARKER: Write out of block quota ... [ 1939.846204] Lustre: DEBUG MARKER: Write... [ 1942.687872] Lustre: DEBUG MARKER: Write out of block quota ... [ 1970.023852] Lustre: DEBUG MARKER: Write... [ 1972.544405] Lustre: DEBUG MARKER: Write out of block quota ... [ 2005.261184] Lustre: DEBUG MARKER: == sanity-quota test 1l: Async writes should not be rejected by quota with root_prj_enable ========================================================== 23:45:32 (1786419932) [ 2044.460127] Lustre: DEBUG MARKER: SKIP: sanity-quota test_2 skipping excluded test 2 [ 2046.571542] Lustre: DEBUG MARKER: == sanity-quota test 3a: Block soft limit (start timer, timer goes off, stop timer) ========================================================== 23:46:13 (1786419973) [ 2094.737495] Lustre: DEBUG MARKER: Write after timer goes off [ 2097.332579] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2170.581976] Lustre: DEBUG MARKER: Write after timer goes off [ 2172.281570] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2250.713956] Lustre: DEBUG MARKER: Write after timer goes off [ 2253.727665] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2317.063940] Lustre: DEBUG MARKER: == sanity-quota test 3b: Quota pools: Block soft limit (start timer, expires, stop timer) ========================================================== 23:50:44 (1786420244) [ 2371.638599] Lustre: DEBUG MARKER: Write after timer goes off [ 2373.415040] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2440.779647] Lustre: DEBUG MARKER: Write after timer goes off [ 2443.033982] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2513.159020] Lustre: DEBUG MARKER: Write after timer goes off [ 2515.836027] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2591.160637] Lustre: DEBUG MARKER: == sanity-quota test 3c: Quota pools: check block soft limit on different pools ========================================================== 23:55:18 (1786420518) [ 2654.245595] Lustre: DEBUG MARKER: Write after timer goes off [ 2656.542471] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2737.534951] Lustre: DEBUG MARKER: SKIP: sanity-quota test_4a skipping excluded test 4a [ 2739.203500] Lustre: DEBUG MARKER: == sanity-quota test 4b: Grace time strings handling ===== 23:57:46 (1786420666) [ 2750.070722] Lustre: DEBUG MARKER: == sanity-quota test 5: Chown [ 2869.146945] Lustre: DEBUG MARKER: == sanity-quota test 6: Test dropping acquire request on master ========================================================== 23:59:56 (1786420796) [ 2907.086149] Lustre: *** cfs_fail_loc=513, val=601*** [ 2907.090043] Lustre: Skipped 3 previous similar messages [ 2908.135700] Lustre: *** cfs_fail_loc=513, val=601*** [ 2908.143134] Lustre: Skipped 6 previous similar messages [ 2909.137962] Lustre: *** cfs_fail_loc=513, val=601*** [ 2909.144557] Lustre: Skipped 13 previous similar messages [ 2909.323286] LustreError: 16519:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1873195072057216 [ 2911.711815] Lustre: *** cfs_fail_loc=513, val=601*** [ 2911.713622] Lustre: Skipped 8 previous similar messages [ 2915.807516] Lustre: *** cfs_fail_loc=513, val=601*** [ 2915.816260] Lustre: Skipped 21 previous similar messages [ 2925.025445] Lustre: 19443:0:(service.c:1612:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff9e783fe1c380 x1873195056336128/t0(0) o4->18d19595-9f92-4aa3-a0c2-93ced4a2b608@192.168.206.6@tcp:259/0 lens 488/448 e 1 to 0 dl 1786420859 ref 2 fl Interpret:/600/0 rc 0/0 job:'dd.60000' uid:60000 gid:60000 projid:0 [ 2925.539091] Lustre: 8435:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786420839/real 1786420839] req@ffff9e770ad86a00 x1873195072057216/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1786420855 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_002.0' uid:0 gid:0 projid:4294967295 [ 2925.566968] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2925.576323] Lustre: *** cfs_fail_loc=513, val=601*** [ 2925.579686] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 2925.582521] Lustre: Skipped 52 previous similar messages [ 2925.596510] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2925.702651] LustreError: 6517:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1873195072066176 [ 2929.631634] LustreError: 16521:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1873195072067328 [ 2929.641319] LustreError: 16521:0:(service.c:2341:ptlrpc_server_handle_req_in()) Skipped 2 previous similar messages [ 2941.911455] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786420855/real 1786420855] req@ffff9e783fe28a80 x1873195072066176/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1786420871 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_007.0' uid:0 gid:0 projid:4294967295 [ 2941.981649] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.007812] Lustre: *** cfs_fail_loc=513, val=601*** [ 2942.009733] Lustre: Skipped 84 previous similar messages [ 2942.034162] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 2942.046622] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2945.085729] LustreError: 16517:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1873195072076672 [ 2946.015371] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786420859/real 1786420859] req@ffff9e7708af5880 x1873195072067200/t0(0) o601->lustre-MDT0000-lwp-MDT0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1786420875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 2946.016730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.062978] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2946.108526] Lustre: lustre-MDT0000: Received new LWP connection from 0@lo, keep former export from same NID [ 2946.129429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2961.378828] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786420875/real 1786420875] req@ffff9e7708e6b100 x1873195072076928/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1786420891 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 2961.383443] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.422536] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2961.456947] Lustre: Skipped 1 previous similar message [ 2961.463710] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 2961.470507] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2961.471939] Lustre: Skipped 1 previous similar message [ 2998.794959] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 00:02:05 (1786420925) [ 3025.063047] Lustre: Failing over lustre-OST0000 [ 3025.299216] Lustre: server umount lustre-OST0000 complete [ 3027.941549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.951435] Lustre: Skipped 1 previous similar message [ 3033.061560] LustreError: 42047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3033.080992] LustreError: 42047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3035.088073] LustreError: 41682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.078462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3037.332172] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 3037.373873] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3038.694530] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3039.258657] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3039.258696] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3039.266353] Lustre: Skipped 1 previous similar message [ 3044.212175] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3052.966696] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3058.616543] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3064.652401] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3072.276372] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3078.447316] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3084.292208] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3106.654700] Lustre: Failing over lustre-OST0000 [ 3106.763258] Lustre: server umount lustre-OST0000 complete [ 3106.818645] LustreError: 41682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.356244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3109.404818] LustreError: 41682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.432397] LustreError: 41682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3114.470900] LustreError: 42047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.493711] LustreError: 42047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3115.759094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3116.063954] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 3116.085819] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3117.020295] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3117.522864] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3117.523594] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3117.547673] Lustre: Skipped 1 previous similar message [ 3122.658128] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3131.314510] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3138.270290] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3145.058261] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3150.873327] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3158.005725] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3163.879286] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3192.798514] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) ========================================================== 00:05:20 (1786421120) [ 3225.297243] Lustre: *** cfs_fail_loc=a02, val=0*** [ 3232.299113] Lustre: Failing over lustre-OST0000 [ 3232.371971] Lustre: server umount lustre-OST0000 complete [ 3233.762730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3233.792746] Lustre: Skipped 1 previous similar message [ 3233.809831] LustreError: 42047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3233.835352] LustreError: 42047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3240.655210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3240.946017] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 3240.971366] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3242.036359] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3242.701284] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3242.717585] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3242.723725] Lustre: Skipped 1 previous similar message [ 3246.625493] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3253.851239] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3258.625789] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3264.020508] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3269.179971] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3273.834918] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3278.853995] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3312.370681] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ========================================================== 00:07:19 (1786421239) [ 3331.284116] LustreError: 97853:0:(qsd_reint.c:488:qsd_reint_main()) cfs_fail_timeout id a03 sleeping for 10000ms [ 3331.308531] LustreError: 97853:0:(qsd_reint.c:488:qsd_reint_main()) Skipped 5 previous similar messages [ 3332.797280] Lustre: Failing over lustre-MDT0000 [ 3333.097522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.103239] Lustre: Skipped 1 previous similar message [ 3333.106928] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.118381] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3333.162809] Lustre: server umount lustre-MDT0000 complete [ 3335.927661] LustreError: 97857:0:(qsd_reint.c:488:qsd_reint_main()) cfs_fail_timeout interrupted [ 3341.787683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.947272] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3342.272238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3342.322300] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3343.843621] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3346.028090] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3347.435975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3347.440447] Lustre: Skipped 1 previous similar message [ 3347.445793] LustreError: 3640:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x2020000:0x0].0x0 (ffff9e783526d700) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3347.557387] Lustre: 97858:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0001: index version for fid [0x200000005:0x100c:0x0] is 0, but index isn't empty (1) [ 3347.581771] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3347.637172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:149 to 0x280000401:193) [ 3347.639177] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:115 to 0x2c0000401:161) [ 3353.348683] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3455.097808] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3460.374392] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3466.136431] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3472.301978] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3478.595169] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3510.309770] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ========================================================== 00:10:37 (1786421437) [ 3532.182668] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3537.121537] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3573.143660] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================================== 00:11:40 (1786421500) [ 3597.916538] Lustre: Failing over lustre-MDT0001 [ 3598.215990] Lustre: server umount lustre-MDT0001 complete [ 3598.319356] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.322858] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3598.322905] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3598.322911] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3598.339720] Lustre: Skipped 4 previous similar messages [ 3608.753971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3609.082733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3609.136108] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 3609.565839] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3613.665730] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3614.185371] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3614.194314] Lustre: Skipped 3 previous similar messages [ 3614.214570] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3614.254610] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3621.702775] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3627.171131] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 3632.927596] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3638.438411] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 3644.178985] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3649.554720] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 3723.985606] Lustre: Failing over lustre-MDT0001 [ 3724.312867] Lustre: server umount lustre-MDT0001 complete [ 3726.819390] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.825452] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3726.825715] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3726.825725] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3726.841953] Lustre: Skipped 2 previous similar messages [ 3732.466160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3733.001539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3733.088550] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 3734.050701] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3737.395740] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3738.103957] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3738.165431] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3746.229769] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3751.849521] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 3757.346871] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3762.778510] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 3767.950949] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3772.893827] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 3852.491871] Lustre: DEBUG MARKER: == sanity-quota test 7f: Quota reintegration automatically ========================================================== 00:16:19 (1786421779) [ 3863.740387] Lustre: *** cfs_fail_loc=a11, val=0*** [ 3870.109775] Lustre: *** cfs_fail_loc=a11, val=0*** [ 3924.964917] Lustre: *** cfs_fail_loc=a11, val=0*** [ 3925.027431] Lustre: 114692:0:(qsd_reint.c:249:qsd_reint_index()) lustre-MDT0001: index version for fid [0x200000005:0x1005:0x0] is 0, but index isn't empty (1) [ 3965.593994] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ==== 00:18:12 (1786421892) [ 4157.267448] Lustre: DEBUG MARKER: == sanity-quota test 9: Block limit larger than 4GB (b10707) ========================================================== 00:21:24 (1786422084) [ 4158.701570] Lustre: DEBUG MARKER: OST0_SIZE: 3605272 required: 4900000 [ 4164.551286] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ======== 00:21:32 (1786422092) [ 4202.813141] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota ======= 00:22:10 (1786422130) [ 4240.936291] Lustre: DEBUG MARKER: == sanity-quota test 12a: Block quota rebalancing ======== 00:22:48 (1786422168) [ 4306.407719] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ======== 00:23:53 (1786422233) [ 4444.333692] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================================== 00:26:11 (1786422371) [ 4500.474877] Lustre: DEBUG MARKER: == sanity-quota test 14: check panic in qmt_site_recalc_cb ========================================================== 00:27:07 (1786422427) [ 4522.662504] Lustre: Failing over lustre-OST0000 [ 4522.877885] Lustre: server umount lustre-OST0000 complete [ 4525.536141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4525.545281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4525.572804] LustreError: 42048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.603641] LustreError: 42048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4535.716809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4536.092260] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4536.136278] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4537.306708] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4537.853495] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4537.856420] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4537.898103] Lustre: Skipped 5 previous similar messages [ 4541.930409] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4577.732932] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ========= 00:28:24 (1786422504) [ 4607.358835] Lustre: DEBUG MARKER: == sanity-quota test 16a: lfs quota should skip the inactive MDT/OST ========================================================== 00:28:54 (1786422534) [ 4626.797769] Lustre: lustre-MDT0001: Client 18d19595-9f92-4aa3-a0c2-93ced4a2b608 (at 192.168.206.6@tcp) reconnecting [ 4649.064310] Lustre: DEBUG MARKER: == sanity-quota test 16b: lfs quota should skip the nonexistent MDT/OST ========================================================== 00:29:36 (1786422576) [ 4650.451294] Lustre: DEBUG MARKER: SKIP: sanity-quota test_16b needs >= 3 MDTs [ 4652.290992] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error == 00:29:39 (1786422579) [ 4670.103470] Lustre: *** cfs_fail_loc=a04, val=37*** [ 4670.107368] LustreError: 116346:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x1 qsd:lustre-OST0001 qtype:usr lqe: ffff9e7836cf9080 id:60000 enforced:1 granted: 0 pending:0 waiting:1032 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 [ 4671.218996] Lustre: *** cfs_fail_loc=a04, val=37*** [ 4671.231260] LustreError: 116344:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x1 qsd:lustre-OST0001 qtype:usr lqe: ffff9e7836cf9080 id:60000 enforced:1 granted: 0 pending:0 waiting:1056 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 [ 4720.064683] Lustre: *** cfs_fail_loc=a04, val=11*** [ 4723.190820] Lustre: *** cfs_fail_loc=a04, val=11*** [ 4723.196018] Lustre: Skipped 1 previous similar message [ 4773.041733] Lustre: *** cfs_fail_loc=a04, val=110*** [ 4827.880718] Lustre: *** cfs_fail_loc=a04, val=107*** [ 4827.887106] Lustre: Skipped 2 previous similar messages [ 4907.818853] Lustre: DEBUG MARKER: == sanity-quota test 18: MDS failover while writing, no watchdog triggered (b14840) ========================================================== 00:33:55 (1786422835) [ 4919.865654] Lustre: DEBUG MARKER: User quota (limit: 200) [ 4924.958361] Lustre: DEBUG MARKER: Write 100M (buffered) ... [ 4932.994469] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4934.920824] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 4937.094901] Lustre: Failing over lustre-MDT0000 [ 4937.450899] Lustre: server umount lustre-MDT0000 complete [ 4938.689096] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107 [ 4938.697628] LustreError: 24115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4938.711148] LustreError: 24115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4956.168290] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786422870/real 1786422870] req@ffff9e783fdf3100 x1873195074502400/t0(0) o400->MGC192.168.206.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786422886 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4956.191883] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4956.199934] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4958.587297] LDISKFS-fs (dm-0): 7 truncates cleaned up [ 4958.590471] LDISKFS-fs (dm-0): recovery complete [ 4958.610282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.376793] LustreError: 3640:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e783d1c8700 x1873195074512512/t0(0) o250->MGC192.168.206.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4966.827638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4966.876897] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4968.348023] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4972.032101] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4972.035954] Lustre: Skipped 1 previous similar message [ 4972.108362] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4972.146714] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:643 to 0x2c0000401:673) [ 4972.147040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:679 to 0x280000401:705) [ 4973.263873] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4983.493599] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4985.242820] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4991.118222] Lustre: DEBUG MARKER: (dd_pid=127500, time=0, timeout=600) [ 5024.442731] Lustre: DEBUG MARKER: User quota (limit: 200) [ 5031.749858] Lustre: DEBUG MARKER: Write 100M (directio) ... [ 5040.760151] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5042.229114] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 5044.297789] Lustre: Failing over lustre-MDT0000 [ 5045.218189] LustreError: 8451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.244412] LustreError: 8451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 31 previous similar messages [ 5046.606702] Lustre: server umount lustre-MDT0000 complete [ 5046.755826] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.764665] Lustre: Skipped 7 previous similar messages [ 5062.111596] Lustre: 3642:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786422976/real 1786422976] req@ffff9e783d1c8a80 x1873195074574080/t0(0) o400->MGC192.168.206.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786422992 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5062.145195] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5069.727160] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5069.728844] LDISKFS-fs (dm-0): recovery complete [ 5069.736917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5073.682195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5073.735762] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5075.101089] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5079.031062] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5079.044395] Lustre: Skipped 3 previous similar messages [ 5079.063533] LustreError: 3640:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-MDT0001: namespace resource [0x200000006:0x10000:0x0].0x0 (ffff9e7707733a00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5079.119467] LustreError: 3640:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages [ 5079.160526] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5079.227201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:707 to 0x280000401:737) [ 5079.229212] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:643 to 0x2c0000401:705) [ 5079.748732] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5089.885219] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5092.197463] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5097.233649] Lustre: DEBUG MARKER: (dd_pid=130028, time=0, timeout=600) [ 5135.441458] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 00:37:42 (1786423062) [ 5188.781376] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 00:38:35 (1786423115) [ 5212.438658] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [ 5222.832862] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [ 5224.370549] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [ 5226.400601] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000 [ 5228.422454] Lustre: DEBUG MARKER: Set quota for 1 times [ 5231.756895] Lustre: DEBUG MARKER: Set quota for 2 times [ 5235.021387] Lustre: DEBUG MARKER: Set quota for 3 times [ 5238.699238] Lustre: DEBUG MARKER: Set quota for 4 times [ 5241.634516] Lustre: DEBUG MARKER: Set quota for 5 times [ 5245.185327] Lustre: DEBUG MARKER: Set quota for 6 times [ 5248.325779] Lustre: DEBUG MARKER: Set quota for 7 times [ 5251.719719] Lustre: DEBUG MARKER: Set quota for 8 times [ 5254.930122] Lustre: DEBUG MARKER: Set quota for 9 times [ 5299.040944] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 00:40:26 (1786423226) [ 5314.532736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5319.539560] Lustre: server umount lustre-MDT0000 complete [ 5319.650172] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5319.685533] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 5323.145828] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423253 with bad export cookie 6066285306415844486 [ 5323.148420] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5323.158101] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5323.533278] Lustre: server umount lustre-MDT0001 complete [ 5337.324428] Lustre: server umount lustre-OST0000 complete [ 5340.196981] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786423254/real 1786423254] req@ffff9e783ca44000 x1873195074772608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786423270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5341.312157] Lustre: server umount lustre-OST0001 complete [ 5356.534836] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 5367.254697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.647964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.604216] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5379.591169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.926847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5383.889618] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5387.071601] Lustre: 154482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5393.531358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5393.945705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.820087] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5404.204944] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 5408.437042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5408.758465] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5412.845251] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:708 to 0x2c0000401:737) [ 5412.864540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:769) [ 5415.140982] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5422.227364] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5425.860962] Lustre: 156357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5446.625430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5446.639316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5446.645208] Lustre: Skipped 3 previous similar messages [ 5448.672035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5448.682564] Lustre: Skipped 2 previous similar messages [ 5451.256575] Lustre: server umount lustre-MDT0000 complete [ 5453.801852] LustreError: 153323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5453.816189] LustreError: 153323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5455.031663] LustreError: 153308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423384 with bad export cookie 6066285306415851780 [ 5455.035737] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5455.046985] LustreError: 153308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5455.315996] Lustre: server umount lustre-MDT0001 complete [ 5469.756725] Lustre: server umount lustre-OST0000 complete [ 5483.181679] Lustre: server umount lustre-OST0001 complete [ 5499.218744] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 5509.259333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.668400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5514.370057] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5523.164648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5528.587398] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5531.387576] Lustre: 160148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5538.113888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5545.020530] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5548.733513] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129) [ 5554.443954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5554.745490] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5554.751505] Lustre: Skipped 2 previous similar messages [ 5555.761445] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:708 to 0x2c0000401:769) [ 5555.779428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:801) [ 5561.768553] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5570.440548] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5574.803771] Lustre: 162018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5585.923945] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 00:45:12 (1786423512) [ 5587.833974] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144 [ 5589.687670] Lustre: DEBUG MARKER: run for 4MB test file [ 5603.193607] Lustre: DEBUG MARKER: User quota (limit: 4 MB) [ 5609.948584] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT [ 5612.067871] Lustre: DEBUG MARKER: Write half of file [ 5614.193579] Lustre: DEBUG MARKER: Write out of block quota ... [ 5616.195851] Lustre: DEBUG MARKER: Step1: done [ 5617.900744] Lustre: DEBUG MARKER: Step2: rewrite should succeed [ 5619.757342] Lustre: DEBUG MARKER: Step2: done [ 5647.081222] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440 [ 5648.853687] Lustre: DEBUG MARKER: run for 40MB test file [ 5660.373797] Lustre: DEBUG MARKER: User quota (limit: 40 MB) [ 5666.313658] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT [ 5668.062505] Lustre: DEBUG MARKER: Write half of file [ 5671.220779] Lustre: DEBUG MARKER: Write out of block quota ... [ 5674.749125] Lustre: DEBUG MARKER: Step1: done [ 5676.191419] Lustre: DEBUG MARKER: Step2: rewrite should succeed [ 5677.972808] Lustre: DEBUG MARKER: Step2: done [ 5729.058894] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 00:47:36 (1786423656) [ 5771.204933] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 00:48:18 (1786423698) [ 5811.053265] Lustre: DEBUG MARKER: Write... [ 5813.577368] Lustre: DEBUG MARKER: Write out of block quota ... [ 5862.805410] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 00:49:50 (1786423790) [ 5868.964752] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 00:49:56 (1786423796) [ 5879.636485] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 00:50:06 (1786423806) [ 5887.673669] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 00:50:15 (1786423815) [ 5895.838575] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 00:50:23 (1786423823) [ 5944.376769] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user [ 6082.879744] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user [ 6221.540806] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 00:55:49 (1786424149) [ 6266.797372] Lustre: DEBUG MARKER: Restart... [ 6270.476385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.500354] Lustre: Skipped 10 previous similar messages [ 6270.509248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6270.524460] Lustre: Skipped 1 previous similar message [ 6275.552808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6275.560756] Lustre: Skipped 6 previous similar messages [ 6276.921556] Lustre: server umount lustre-MDT0000 complete [ 6280.523249] LustreError: 158990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424210 with bad export cookie 6066285306415854083 [ 6280.523807] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6280.537315] LustreError: 158990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6280.682964] LustreError: 183276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6280.685951] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6280.713656] LustreError: 183276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6280.726653] Lustre: Skipped 1 previous similar message [ 6280.883752] Lustre: server umount lustre-MDT0001 complete [ 6287.122490] Lustre: server umount lustre-OST0000 complete [ 6291.005663] Lustre: server umount lustre-OST0001 complete [ 6307.107468] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 6317.309989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6317.991864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.473477] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6330.110879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.420724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6334.312421] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6336.879996] Lustre: 186359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6342.984926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6349.179943] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6355.640826] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161) [ 6356.900275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6357.310709] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6357.324774] Lustre: Skipped 1 previous similar message [ 6362.439103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:812 to 0x280000401:833) [ 6362.446084] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:778 to 0x2c0000401:801) [ 6363.976828] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6370.630277] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6373.657967] Lustre: 188232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6423.591664] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 00:59:11 (1786424351) [ 6469.909525] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 00:59:57 (1786424397) [ 8150.801606] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 01:27:58 (1786426078) [ 8163.296058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8163.306785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8163.320704] Lustre: Skipped 5 previous similar messages [ 8163.327691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8164.832202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8167.855356] Lustre: server umount lustre-MDT0000 complete [ 8169.951809] LustreError: 186724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8169.967365] LustreError: 186724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 8171.965319] LustreError: 185198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426101 with bad export cookie 6066285306415863204 [ 8171.969486] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8171.976271] LustreError: 185198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8172.417186] Lustre: server umount lustre-MDT0001 complete [ 8187.216483] Lustre: server umount lustre-OST0000 complete [ 8190.432721] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786426104/real 1786426104] req@ffff9e77093cd880 x1873195078779520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786426120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8190.490988] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8190.504070] Lustre: Skipped 3 previous similar messages [ 8191.791739] Lustre: server umount lustre-OST0001 complete [ 8210.054888] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 8221.147637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.519617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.889668] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8234.490168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8234.869368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8238.900096] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8242.023558] Lustre: 196954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8248.207770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8248.560113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8254.517067] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8259.812176] LustreError: 197307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8259.832600] LustreError: 197307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8259.835503] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193) [ 8264.086456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8266.346264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5835 to 0x280000401:5857) [ 8266.356156] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5803 to 0x2c0000401:5825) [ 8271.272591] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8279.128363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8282.962805] Lustre: 198827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8309.733947] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 01:30:37 (1786426237) [ 8338.346621] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 01:31:05 (1786426265) [ 8362.882373] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 01:31:30 (1786426290) [ 8394.244934] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 01:32:01 (1786426321) [ 8430.379710] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 01:32:37 (1786426357) [ 8510.587966] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 01:33:57 (1786426437) [ 8546.681654] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 01:34:33 (1786426473) [ 8615.976332] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 01:35:43 (1786426543) [ 8621.624073] Lustre: *** cfs_fail_loc=a09, val=0*** [ 8621.628104] Lustre: Skipped 1 previous similar message [ 8623.640389] Lustre: *** cfs_fail_loc=a09, val=0*** [ 8623.644566] Lustre: Skipped 91 previous similar messages [ 8627.655256] Lustre: *** cfs_fail_loc=a09, val=0*** [ 8627.657877] Lustre: Skipped 173 previous similar messages [ 8635.674293] Lustre: *** cfs_fail_loc=a09, val=0*** [ 8635.680067] Lustre: Skipped 361 previous similar messages [ 8651.678299] Lustre: *** cfs_fail_loc=a09, val=0*** [ 8651.680208] Lustre: Skipped 727 previous similar messages [ 8683.689623] Lustre: *** cfs_fail_loc=a09, val=0*** [ 8683.698599] Lustre: Skipped 1447 previous similar messages [ 8747.757172] Lustre: *** cfs_fail_loc=a09, val=0*** [ 8747.763059] Lustre: Skipped 2743 previous similar messages [ 9051.747304] Lustre: *** cfs_fail_loc=a08, val=0*** [ 9051.752894] Lustre: Skipped 2451 previous similar messages [ 9051.768222] Lustre: *** cfs_fail_loc=a08, val=0*** [ 9051.771909] Lustre: Skipped 2 previous similar messages [ 9171.939108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9171.943915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9171.956316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9171.959979] Lustre: Skipped 3 previous similar messages [ 9172.962415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9178.080682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9178.098180] Lustre: Skipped 6 previous similar messages [ 9178.506779] Lustre: server umount lustre-MDT0000 complete [ 9182.874319] LustreError: 195791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786427112 with bad export cookie 6066285306417641904 [ 9182.876910] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9182.885381] LustreError: 195791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9183.201775] LustreError: 197631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9183.203207] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.206022] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9183.223966] LustreError: 197631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9183.242940] Lustre: Skipped 4 previous similar messages [ 9187.233792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9187.236656] Lustre: Skipped 2 previous similar messages [ 9189.429767] Lustre: server umount lustre-MDT0001 complete [ 9194.347704] Lustre: server umount lustre-OST0000 complete [ 9197.969823] Lustre: server umount lustre-OST0001 complete [ 9204.370751] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 9212.584264] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 9258.902903] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 9269.844948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9270.237288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9270.303415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9270.376261] Lustre: lustre-MDT0000: new disk, initializing [ 9270.471318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9270.482087] Lustre: Skipped 1 previous similar message [ 9270.506477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9276.399272] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9287.765278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9287.899952] Lustre: 215530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9287.937208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9287.941378] Lustre: Skipped 1 previous similar message [ 9288.071220] Lustre: lustre-MDT0001: new disk, initializing [ 9288.177589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9288.236645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9288.248337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9293.456808] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9298.607567] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9308.734699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9309.176421] Lustre: lustre-OST0000: new disk, initializing [ 9309.180575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9309.184866] Lustre: 217164:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9309.307462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9310.545921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9310.572477] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9310.750277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9318.519525] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9333.370885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9333.497783] Lustre: lustre-OST0001: new disk, initializing [ 9333.502450] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9333.508235] Lustre: 218033:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9333.593593] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9334.787717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9334.804279] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9334.927104] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9341.479164] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9352.515516] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9357.250476] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9387.054507] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 01:48:34 (1786427314) [ 9394.010675] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 01:48:41 (1786427321) [ 9428.224635] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 01:49:15 (1786427355) [ 9486.063732] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 01:50:13 (1786427413) [ 9511.620793] Lustre: DEBUG MARKER: rename directory return 255 [ 9554.130152] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 01:51:20 (1786427480) [ 9578.681641] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 01:51:46 (1786427506) [ 9617.957161] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 01:52:25 (1786427545) [ 9746.577400] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 01:54:33 (1786427673) [ 9776.515102] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 01:55:03 (1786427703) [ 9807.574953] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 01:55:34 (1786427734) [ 9953.280826] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 01:58:00 (1786427880) [ 9958.880383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9958.889901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9958.904890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9963.948979] Lustre: server umount lustre-MDT0000 complete [ 9964.526233] LustreError: 216407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9964.556115] LustreError: 216407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9968.806956] LustreError: 215523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786427898 with bad export cookie 6066285306418014073 [ 9968.810921] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9968.842846] LustreError: 215523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9969.288309] Lustre: server umount lustre-MDT0001 complete [ 9984.980427] Lustre: server umount lustre-OST0000 complete [ 9985.891514] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786427899/real 1786427899] req@ffff9e7837163100 x1873195085867520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786427915 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9985.954494] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9985.977771] Lustre: Skipped 3 previous similar messages [ 9989.092659] Lustre: 3641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786427903/real 1786427903] req@ffff9e7837161f80 x1873195085867776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786427919 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9990.111814] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786427904/real 1786427904] req@ffff9e7837140700 x1873195085868032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786427920 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9990.890975] Lustre: server umount lustre-OST0001 complete [10012.888372] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [10022.789630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10023.175745] LustreError: 235006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10023.278252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10023.299619] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already [10027.632600] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10028.514747] LustreError: 235007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10036.517371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10036.772060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10036.800421] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already [10041.053443] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10043.921154] Lustre: 236151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10051.337627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10051.709317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10051.717594] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already [10058.475931] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10061.923788] LustreError: 236505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10061.951230] LustreError: 236505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10067.181095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10067.373642] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10067.382524] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already [10068.400389] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65) [10068.403351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65) [10074.130907] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10082.629925] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10086.549476] Lustre: 238025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10093.637345] LustreError: 236276:0:(osd_handler.c:3426:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95 [10098.150715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10098.158882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10098.172596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10098.179505] Lustre: Skipped 4 previous similar messages [10104.068393] Lustre: server umount lustre-MDT0000 complete [10104.291413] LustreError: 236276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10104.327597] LustreError: 236276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10107.947190] LustreError: 234988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786428037 with bad export cookie 6066285306418062968 [10107.956422] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10107.963072] LustreError: 234988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10108.295888] Lustre: server umount lustre-MDT0001 complete [10114.017680] LustreError: 3641:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9e770c4e9500 x1873195085921664/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [10114.030838] LustreError: 3644:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8 qsd:lustre-OST0000 qtype:usr lqe: ffff9e770e39f500 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1516 qunit:0 qtune:0 edquot:0 default:no revoke:0 [10114.039164] LustreError: 3644:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message [10121.896967] Lustre: server umount lustre-OST0000 complete [10130.401195] LustreError: 3641:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9e7813530a80 x1873195085922432/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [10130.435623] LustreError: 3641:0:(client.c:1381:ptlrpc_import_delay_req()) Skipped 1 previous similar message [10130.446608] LustreError: 3641:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8 qsd:lustre-OST0001 qtype:usr lqe: ffff9e770e3ba480 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1516 qunit:0 qtune:0 edquot:0 default:no revoke:0 [10130.461439] LustreError: 3641:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message [10135.859331] Lustre: server umount lustre-OST0001 complete [10158.803771] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [10170.817732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10171.318248] LustreError: 240426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10171.385880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10176.014501] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10185.386896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10190.484839] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10193.289536] Lustre: 241571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10200.504689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10207.247250] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10216.182889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:97) [10216.687403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10216.814737] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10216.819429] Lustre: Skipped 2 previous similar messages [10222.063290] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10222.542229] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10230.045575] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10233.466055] Lustre: 243446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10257.780877] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 02:03:05 (1786428185) [10304.342879] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61 [10306.055225] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 02:03:53 (1786428233) [10326.115532] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63 [10327.952836] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 02:04:15 (1786428255) [10362.477384] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65 [10364.316968] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 02:04:51 (1786428291) [10398.154873] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 02:05:25 (1786428325) [10409.381628] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [10415.593930] Lustre: 249911:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1002:0x0] is 0, but index isn't empty (1) [10415.617403] Lustre: 249911:0:(qsd_reint.c:249:qsd_reint_index()) Skipped 2 previous similar messages [10420.056919] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [10425.205417] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [10428.576786] Lustre: DEBUG MARKER: Write... [10449.495181] LustreError: 251257:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1 [10455.307966] Lustre: DEBUG MARKER: Write... [10470.700465] Lustre: DEBUG MARKER: Write... [10544.752460] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 02:07:52 (1786428472) [10570.135368] LustreError: 255275:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1 [10618.621738] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 02:09:05 (1786428545) [10643.529855] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [10645.481902] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [10726.017207] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 02:10:53 (1786428653) [10773.264744] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 02:11:40 (1786428700) [10811.587354] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 02:12:18 (1786428738) [10822.924274] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB) [10847.604385] Lustre: DEBUG MARKER: Write... [10849.679396] Lustre: DEBUG MARKER: Write out of block quota ... [10942.142199] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 02:14:29 (1786428869) [10954.458201] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB) [11035.705289] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 02:16:02 (1786428962) [11047.752898] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB) [11066.839950] Lustre: DEBUG MARKER: Write... [11068.751859] Lustre: DEBUG MARKER: Write out of block quota ... [11123.897678] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 02:17:30 (1786429050) [11151.473965] Lustre: DEBUG MARKER: set to use default quota [11153.386364] Lustre: DEBUG MARKER: set default quota [11155.029640] Lustre: DEBUG MARKER: get default quota [11161.591153] Lustre: DEBUG MARKER: Test not out of quota [11165.073867] Lustre: DEBUG MARKER: Test out of quota [11178.074844] Lustre: DEBUG MARKER: Increase default quota [11203.542334] Lustre: DEBUG MARKER: Set quota to override default quota [11203.635475] LustreError: 240423:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1787033933 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [11215.688549] Lustre: DEBUG MARKER: Set to use default quota again [11235.928707] Lustre: DEBUG MARKER: Cleanup [11291.588086] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 02:20:18 (1786429218) [11313.164431] Lustre: DEBUG MARKER: set default quota for qpool1 [11314.899460] Lustre: DEBUG MARKER: Write from user that hasn't lqe [11358.855452] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 02:21:26 (1786429286) [11431.497975] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 02:22:39 (1786429359) [11495.434705] Lustre: DEBUG MARKER: Write... [11498.291677] Lustre: DEBUG MARKER: Write out of block quota ... [11593.868534] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 02:25:21 (1786429521) [11626.158465] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 02:25:53 (1786429553) [11634.217747] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 02:26:01 (1786429561) [11671.058512] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 02:26:38 (1786429598) [11706.974336] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 02:27:14 (1786429634) [11729.364805] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 02:27:36 (1786429656) [11755.787714] Lustre: *** cfs_fail_loc=a06, val=0*** [11756.151619] LustreError: 240440:0:(qmt_lock.c:479:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11 [11756.151619] qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:25600 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [11761.092943] Lustre: Failing over lustre-OST0001 [11761.136016] Lustre: lustre-OST0001: Not available for connect from 192.168.206.6@tcp (stopping) [11761.142958] Lustre: Skipped 4 previous similar messages [11761.415545] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11761.423435] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11761.433935] Lustre: Skipped 3 previous similar messages [11763.167410] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11763.172880] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11763.184804] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11763.192766] Lustre: Skipped 1 previous similar message [11763.434274] Lustre: server umount lustre-OST0001 complete [11766.237986] LustreError: 242708:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11766.262088] LustreError: 242708:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11772.019699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11772.277171] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [11772.299489] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect [11773.407287] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect [11774.276780] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [11774.279761] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [11774.295356] Lustre: Skipped 3 previous similar messages [11777.491665] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11824.877603] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 02:29:12 (1786429752) [11835.324588] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [11842.327291] LustreError: 294615:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms [11848.183037] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.6@tcp (stopping) [11848.192101] Lustre: Skipped 1 previous similar message [11849.186309] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11849.213428] Lustre: Skipped 3 previous similar messages [11852.415112] LustreError: 294615:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake [11852.843930] Lustre: server umount lustre-MDT0000 complete [11853.274352] LustreError: 240422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11853.308214] LustreError: 240422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11862.721141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11862.877212] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11863.249193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11863.315794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:112 to 0x280000401:129) [11863.318093] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:129) [11868.147802] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11868.648805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11868.666531] Lustre: Skipped 1 previous similar message [11895.474508] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 02:30:22 (1786429822) [11901.609305] LustreError: 241175:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0001: Delaying reintegration for qtype:2 until pending updates are flushed. [11916.676873] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 02:30:44 (1786429844) [11938.200611] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 02:31:05 (1786429865) [11982.076535] Lustre: *** cfs_fail_loc=a08, val=0*** [11982.088691] Lustre: Skipped 3926 previous similar messages [11982.102943] Lustre: *** cfs_fail_loc=a08, val=0*** [12068.740972] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 02:33:16 (1786429996) [12096.187341] LustreError: 244644:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:3072 soft:0 granted:3072 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [12146.010962] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 02:34:33 (1786430073) [12209.918536] LustreError: 243901:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1787034939 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [12357.050955] LustreError: 240422:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1787035086 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [12501.980694] LustreError: 278094:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:16384 time:1787035231 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [12613.549464] Lustre: DEBUG MARKER: == sanity-quota test 87: lfs quota -a should print default quota setting ========================================================== 02:42:20 (1786430540) [12620.026675] Lustre: *** cfs_fail_loc=a09, val=0*** [12620.028781] Lustre: Skipped 1 previous similar message [12677.599885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12677.600935] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12677.606901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12677.618934] Lustre: Skipped 3 previous similar messages [12677.623067] Lustre: Skipped 7 previous similar messages [12681.437948] Lustre: server umount lustre-MDT0000 complete [12682.706642] LustreError: 240423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12682.728809] LustreError: 240423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [12687.827395] LustreError: 278094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12688.769804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12688.919984] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12689.265718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12689.339943] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [12689.341624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [12693.149665] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12694.501423] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12694.508759] Lustre: Skipped 2 previous similar messages [12694.518217] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [12706.396874] Lustre: DEBUG MARKER: == sanity-quota test 88: Writing over quota should not hang ========================================================== 02:43:54 (1786430634) [12707.895854] Lustre: DEBUG MARKER: SKIP: sanity-quota test_88 require client with >4k pages [12709.703374] Lustre: DEBUG MARKER: == sanity-quota test 89: Show default quota with squash_uid ========================================================== 02:43:56 (1786430636) [12730.379756] Lustre: DEBUG MARKER: == sanity-quota test 90a: lfs quota should work without mount point ========================================================== 02:44:17 (1786430657) [12747.002194] Lustre: DEBUG MARKER: == sanity-quota test 90b: lfs quota should work with multiple mount points ========================================================== 02:44:34 (1786430674) [12768.158207] Lustre: DEBUG MARKER: == sanity-quota test 91: new quota index files in quota_master ========================================================== 02:44:55 (1786430695) [12786.663995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12786.666041] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12786.674678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12786.703323] Lustre: Skipped 2 previous similar messages [12790.230403] Lustre: server umount lustre-MDT0000 complete [12791.779384] LustreError: 243901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12791.795805] LustreError: 243901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [12793.530557] LustreError: 252148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786430723 with bad export cookie 6066285306419167505 [12793.537593] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12793.540941] LustreError: 252148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12796.897836] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12796.901161] LustreError: 309697:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9e77098cd500 x1873195089639808/t0(0) o101->lustre-MDT0000-lwp-MDT0001@0@lo:23/10 lens 456/496 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0 projid:4294967295 [12796.912490] Lustre: Skipped 1 previous similar message [12796.914763] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12796.936438] LustreError: 309697:0:(client.c:1381:ptlrpc_import_delay_req()) Skipped 1 previous similar message [12796.960753] Lustre: Skipped 4 previous similar messages [12798.945981] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12798.958848] Lustre: Skipped 1 previous similar message [12799.891885] Lustre: server umount lustre-MDT0001 complete [12809.344858] Lustre: server umount lustre-OST0000 complete [12819.263240] Lustre: server umount lustre-OST0001 complete [12827.551892] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [12834.766414] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [12869.201318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12869.403808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12869.429995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12869.495172] Lustre: lustre-MDT0000: new disk, initializing [12869.570990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12869.597727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12873.200418] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12881.413214] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12887.326715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12887.589394] Lustre: lustre-OST0000: new disk, initializing [12887.592921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12887.598520] Lustre: Skipped 1 previous similar message [12887.604641] Lustre: 313310:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12887.650591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12888.747289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12888.758574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12888.793161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12893.900184] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12902.711414] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12910.479520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12910.634384] Lustre: lustre-OST0001: new disk, initializing [12910.637876] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12910.642956] Lustre: 314367:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12910.694587] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12912.700913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [12912.710365] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [12912.737910] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [12918.006518] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12929.370776] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12943.328220] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12943.347219] Lustre: Skipped 2 previous similar messages [12943.358605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12943.368193] Lustre: Skipped 1 previous similar message [12943.989927] Lustre: server umount lustre-MDT0000 complete [12954.535200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12954.663689] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12954.960695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12959.510889] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12963.304508] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12963.314923] Lustre: Skipped 3 previous similar messages [12967.725074] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12969.538380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12971.333625] LustreError: 316045:0:(qmt_entry.c:1174:qmt_map_lge_idx()) qmt: cannot map ostidx 1, num_used 1: rc = -22 [12973.790264] LustreError: 316045:0:(qmt_entry.c:1174:qmt_map_lge_idx()) qmt: cannot map ostidx 1, num_used 1: rc = -22 [12977.522617] Lustre: server umount lustre-MDT0000 complete [12984.025241] LustreError: 312304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786430913 with bad export cookie 6066285306419170172 [12984.051274] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12993.673636] Lustre: server umount lustre-OST0000 complete [12994.535019] Lustre: 3641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786430908/real 1786430908] req@ffff9e77090e4a80 x1873195089699200/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786430924 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12994.573920] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12997.426246] Lustre: server umount lustre-OST0001 complete [13015.101415] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [13022.365627] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [13067.522828] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [13076.154990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13076.356718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13076.376757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13076.430857] Lustre: lustre-MDT0000: new disk, initializing [13076.486678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13076.502985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13080.690388] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13090.692105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13090.832497] Lustre: 320860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13090.878821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13090.883991] Lustre: Skipped 1 previous similar message [13090.994282] Lustre: lustre-MDT0001: new disk, initializing [13091.084168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13091.091909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13095.618756] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13100.190620] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13106.415162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13106.615899] Lustre: lustre-OST0000: new disk, initializing [13106.620648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13106.627317] Lustre: 322490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13108.643279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13108.669150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13108.761046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13112.655565] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13123.006129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13123.178486] Lustre: lustre-OST0001: new disk, initializing [13123.183657] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13123.191801] Lustre: 323362:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13123.311877] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13123.326965] Lustre: Skipped 2 previous similar messages [13124.643141] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13124.739397] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13130.182506] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13139.927473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13143.784833] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [13151.581639] Lustre: DEBUG MARKER: == sanity-quota test 92: Cannot set inode limit with Quota Pools ========================================================== 02:51:18 (1786431078) [13170.950751] Lustre: DEBUG MARKER: == sanity-quota test 93: update projid while client write to OST ========================================================== 02:51:38 (1786431098) [13194.874906] Lustre: *** cfs_fail_loc=170c, val=0*** [13244.475316] Lustre: DEBUG MARKER: == sanity-quota test 94: lfs quota all respects nodemap offset ========================================================== 02:52:51 (1786431171) [13280.735913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13280.742886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13280.762924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13284.879138] Lustre: server umount lustre-MDT0000 complete [13288.398689] LustreError: 320865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13288.413344] LustreError: 320865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13294.940049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13295.060597] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13295.254490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13295.299874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [13300.482635] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13300.705478] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [13300.736546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13300.749571] Lustre: Skipped 2 previous similar messages [13310.948440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13310.961060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13310.980238] Lustre: Skipped 6 previous similar messages [13310.998600] Lustre: Skipped 8 previous similar messages [13315.515961] Lustre: server umount lustre-MDT0000 complete [13316.079000] LustreError: 320867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13316.114719] LustreError: 320867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [13324.380029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13324.512935] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13324.777708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [13329.912281] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [13329.947280] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [13329.966672] Lustre: Skipped 5 previous similar messages [13330.030226] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13337.286374] Lustre: DEBUG MARKER: == sanity-quota test 95a: Correct limits for squashed root ========================================================== 02:54:24 (1786431264) [13361.362407] Lustre: DEBUG MARKER: == sanity-quota test 95b: df respects squashed project id ========================================================== 02:54:48 (1786431288) [13395.444878] Lustre: DEBUG MARKER: == sanity-quota test 96: quota grant should be released when big files are deleted ========================================================== 02:55:22 (1786431322) [13396.826232] Lustre: DEBUG MARKER: SKIP: sanity-quota test_96 OST is too small, skip the test [13398.269166] Lustre: DEBUG MARKER: == sanity-quota test 97a: LQA control commands =========== 02:55:25 (1786431325) [13422.004983] LustreError: 334133:0:(qmt_lqa.c:79:qmt_lqa_destroy()) lustre-QMT0000: cannot destroy lqa-dt-lqa1: rc = -2 [13422.011232] LustreError: 334133:0:(qmt_lqa.c:84:qmt_lqa_destroy()) lustre-QMT0000: cannot destroy lqa-md-lqa1: rc = -2 [13423.615356] Lustre: DEBUG MARKER: == sanity-quota test 97b: Check LQA internals ============ 02:55:50 (1786431350) [13443.142716] LustreError: 335028:0:(qmt_lqa.c:250:qmt_lqa_insert_range()) lustre-QMT0000: LQA range 15-31 partially overlaps with existing range 10-19: rc = -34 [13443.159386] LustreError: 335028:0:(qmt_lqa.c:669:qmt_lqa_add()) lustre-QMT0000: lqa:lqa1 can't add range 15:31: rc = -17 [13446.302234] LustreError: 335224:0:(qmt_lqa.c:825:qmt_lqa_remove()) lustre-QMT0000: lqa-md-lqa1 cannot remove range 10:19: rc = -2 [13455.507966] Lustre: DEBUG MARKER: adding 50 LQA ranges took 1s [13459.816874] Lustre: DEBUG MARKER: Removing 50 LQA ranges took 1s [13467.686571] Lustre: DEBUG MARKER: == sanity-quota test 97c: LQA lfs quota commands and MDS restart consistency ========================================================== 02:56:34 (1786431394) [13471.682645] Lustre: Failing over lustre-MDT0000 [13472.171796] Lustre: server umount lustre-MDT0000 complete [13472.750544] LustreError: 320865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13472.764928] LustreError: 320865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [13473.258624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13473.274498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13481.510259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13481.794271] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13482.258894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13482.274833] Lustre: Skipped 1 previous similar message [13482.352338] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13482.966646] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13487.605390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13487.643799] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [13487.724567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:97) [13487.846659] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13493.084978] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [13500.845516] Lustre: DEBUG MARKER: == sanity-quota test 97d: LQA disk persistence using standard commands ========================================================== 02:57:07 (1786431427) [13513.447875] Lustre: Failing over lustre-MDT0000 [13513.711730] Lustre: server umount lustre-MDT0000 complete [13521.473379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13521.603320] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13521.926671] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13523.928127] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13526.128773] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13527.028887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13527.039184] Lustre: Skipped 3 previous similar messages [13527.065074] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [13527.126353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:129) [13531.140845] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [13561.774979] Lustre: DEBUG MARKER: == sanity-quota test 98: Verify MDT retry on -EDQUOT during mkdir ========================================================== 02:58:09 (1786431489) [13566.962763] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13568.346715] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13573.748786] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13575.254745] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [13581.350297] Lustre: *** cfs_fail_loc=a02, val=1*** [13581.353690] LustreError: 320865:0:(lod_object.c:6285:lod_declare_create()) lustre-MDT0000-mdtlov: Injecting -EDQUOT for directory create on MDT0000 (fail_val=1): rc = -122 [13588.868229] Lustre: DEBUG MARKER: == sanity-quota test 300: inode quota with MDT directory migration at 80% limit ========================================================== 02:58:36 (1786431516) [13605.625027] Lustre: DEBUG MARKER: Creating 819 files as quota_usr ... [13618.171254] Lustre: DEBUG MARKER: Migrating directory from MDT0 to MDT1 ... [13673.245710] Lustre: DEBUG MARKER: Migration completed successfully [13705.905935] Lustre: DEBUG MARKER: == sanity-quota test complete, duration 13422 sec ======== 03:00:33 (1786431633) [13707.307185] Lustre: DEBUG MARKER: === sanity-quota: start cleanup 03:00:34 (1786431634) === [13710.413901] Lustre: DEBUG MARKER: === sanity-quota: finish cleanup 03:00:37 (1786431637) === [13716.449054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13716.450863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13716.452378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13716.452384] Lustre: Skipped 1 previous similar message [13716.479911] Lustre: Skipped 9 previous similar messages [13719.966302] Lustre: server umount lustre-MDT0000 complete [13721.571715] LustreError: 320867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13721.600871] LustreError: 320867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [13727.070820] LustreError: 324831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786431656 with bad export cookie 6066285306419176703 [13727.077727] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13727.083970] LustreError: 324831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13727.315334] Lustre: server umount lustre-MDT0001 complete [13747.167426] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786431661/real 1786431661] req@ffff9e7837085180 x1873195091776256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786431677 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13748.455467] Lustre: server umount lustre-OST0000 complete [13749.024092] Lustre: 3641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786431662/real 1786431662] req@ffff9e7837086300 x1873195091776512/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786431678 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13752.353236] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786431666/real 1786431666] req@ffff9e7837086a00 x1873195091776768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786431682 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13754.847552] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786431668/real 1786431668] req@ffff9e783feae300 x1873195091777152/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786431684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13757.255180] Lustre: server umount lustre-OST0001 complete [13774.648662] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [13777.405488] Key type lgssc unregistered [13777.671338] LNet: 345782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13777.691639] LNetError: 345782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13777.711513] LNet: Removed LNI 192.168.206.106@tcp [13778.553243] Key type .llcrypt unregistered [13778.555492] Key type ._llcrypt unregistered