[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 529165398 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003408] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005028] kvm-guest: setup PV IPIs [ 0.008492] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010028] pid_max: default: 32768 minimum: 301 [ 0.011172] LSM: Security Framework initializing [ 0.012072] Yama: becoming mindful. [ 0.013071] SELinux: Initializing. [ 0.015102] *** VALIDATE selinux *** [ 0.025342] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031523] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033034] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035156] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036158] *** VALIDATE tmpfs *** [ 0.038399] *** VALIDATE proc *** [ 0.040198] *** VALIDATE cgroup *** [ 0.041015] *** VALIDATE cgroup2 *** [ 0.043052] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.045164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.046020] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.047055] Spectre V2 : User space: Vulnerable [ 0.048017] Speculative Store Bypass: Vulnerable [ 0.051210] debug: unmapping init [mem 0xffffffff9e259000-0xffffffff9e260fff] [ 0.054229] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.055909] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.056033] ... version: 2 [ 0.057022] ... bit width: 48 [ 0.058014] ... generic registers: 4 [ 0.059020] ... value mask: 0000ffffffffffff [ 0.060016] ... max period: 00007fffffffffff [ 0.061021] ... fixed-purpose events: 3 [ 0.062021] ... event mask: 000000070000000f [ 0.063386] rcu: Hierarchical SRCU implementation. [ 0.065833] smp: Bringing up secondary CPUs ... [ 0.066740] x86: Booting SMP configuration: [ 0.067040] .... node #0, CPUs: #1 #2 #3 [ 0.071138] smp: Brought up 1 node, 4 CPUs [ 0.073021] smpboot: Max logical packages: 1 [ 0.074022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.153125] node 0 deferred pages initialised in 76ms [ 0.158135] devtmpfs: initialized [ 0.159454] x86/mm: Memory block size: 128MB [ 0.164501] gcov: version magic: 0x41383552 [ 0.170435] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.171143] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.172607] pinctrl core: initialized pinctrl subsystem [ 0.175368] [ 0.176010] ************************************************************* [ 0.179021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181015] ** ** [ 0.184025] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.187023] ** ** [ 0.189019] ** This means that this kernel is built to expose internal ** [ 0.192020] ** IOMMU data structures, which may compromise security on ** [ 0.194024] ** your system. ** [ 0.197023] ** ** [ 0.201020] ** If you see this message and you are not debugging the ** [ 0.203028] ** kernel, report this immediately to your vendor! ** [ 0.207022] ** ** [ 0.210020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.214025] ************************************************************* [ 0.217775] NET: Registered protocol family 16 [ 0.219553] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.224092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.228083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.233158] cpuidle: using governor menu [ 0.234953] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.235597] PCI: Using configuration type 1 for base access [ 0.236138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.244182] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.248035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.254041] cryptd: max_cpu_qlen set to 1000 [ 0.258398] ACPI: Added _OSI(Module Device) [ 0.260021] ACPI: Added _OSI(Processor Device) [ 0.262029] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.264020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.269477] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.278566] ACPI: Interpreter enabled [ 0.281074] ACPI: PM: (supports S0 S3 S4 S5) [ 0.283025] ACPI: Using IOAPIC for interrupt routing [ 0.285229] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.289485] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.300544] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.305060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.309026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.314115] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.322572] acpiphp: Slot [2] registered [ 0.324170] acpiphp: Slot [5] registered [ 0.326158] acpiphp: Slot [6] registered [ 0.327138] acpiphp: Slot [7] registered [ 0.329159] acpiphp: Slot [8] registered [ 0.330186] acpiphp: Slot [9] registered [ 0.331154] acpiphp: Slot [10] registered [ 0.333150] acpiphp: Slot [3] registered [ 0.334117] acpiphp: Slot [4] registered [ 0.336176] acpiphp: Slot [11] registered [ 0.338138] acpiphp: Slot [12] registered [ 0.340133] acpiphp: Slot [13] registered [ 0.341149] acpiphp: Slot [14] registered [ 0.343143] acpiphp: Slot [15] registered [ 0.344115] acpiphp: Slot [16] registered [ 0.346184] acpiphp: Slot [17] registered [ 0.349136] acpiphp: Slot [18] registered [ 0.351176] acpiphp: Slot [19] registered [ 0.352182] acpiphp: Slot [20] registered [ 0.354125] acpiphp: Slot [21] registered [ 0.356200] acpiphp: Slot [22] registered [ 0.358173] acpiphp: Slot [23] registered [ 0.360129] acpiphp: Slot [24] registered [ 0.362149] acpiphp: Slot [25] registered [ 0.363133] acpiphp: Slot [26] registered [ 0.365141] acpiphp: Slot [27] registered [ 0.367153] acpiphp: Slot [28] registered [ 0.369134] acpiphp: Slot [29] registered [ 0.370147] acpiphp: Slot [30] registered [ 0.372192] acpiphp: Slot [31] registered [ 0.374084] PCI host bridge to bus 0000:00 [ 0.376028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.378035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.381033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.384050] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.387051] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.390038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.392224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.396135] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.399481] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.409819] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.415583] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.418042] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.421023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.423027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.427732] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.430782] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.436073] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.440900] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.447017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.462019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.468033] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.475930] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.486027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.495018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.516051] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.527420] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.539033] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.555031] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.586026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.596811] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.603028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.614025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.637028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.648523] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.660022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.671032] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.693023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.704299] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.712020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.720018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.742019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.753943] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.760023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.769025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.800018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.819560] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.822367] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.825426] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.828554] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.831335] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.838067] iommu: Default domain type: Passthrough [ 0.839481] SCSI subsystem initialized [ 0.840280] ACPI: bus type USB registered [ 0.842156] usbcore: registered new interface driver usbfs [ 0.845108] usbcore: registered new interface driver hub [ 0.847156] usbcore: registered new device driver usb [ 0.849265] pps_core: LinuxPPS API ver. 1 registered [ 0.852013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.856092] PTP clock support registered [ 0.858189] EDAC MC: Ver: 3.0.0 [ 0.861139] PCI: Using ACPI for IRQ routing [ 0.862596] NetLabel: Initializing [ 0.864021] NetLabel: domain hash size = 128 [ 0.865013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.867449] NetLabel: unlabeled traffic allowed by default [ 0.876195] vgaarb: loaded [ 0.877303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.879017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.886055] clocksource: Switched to clocksource kvm-clock [ 1.013166] VFS: Disk quotas dquot_6.6.0 [ 1.014926] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.016924] *** VALIDATE ramfs *** [ 1.017909] *** VALIDATE hugetlbfs *** [ 1.019216] pnp: PnP ACPI init [ 1.021518] pnp: PnP ACPI: found 6 devices [ 1.054178] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.058286] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.060990] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.063606] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.066641] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.069499] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.072682] NET: Registered protocol family 2 [ 1.075185] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.081421] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.085611] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.091483] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.095528] TCP: Hash tables configured (established 65536 bind 65536) [ 1.098705] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.102295] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.106609] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.110429] NET: Registered protocol family 1 [ 1.114311] RPC: Registered named UNIX socket transport module. [ 1.117217] RPC: Registered udp transport module. [ 1.120170] RPC: Registered tcp transport module. [ 1.122356] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.126215] NET: Registered protocol family 44 [ 1.128283] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.130589] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.135821] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.138359] PCI: CLS 0 bytes, default 64 [ 1.139888] Unpacking initramfs... [ 2.582906] debug: unmapping init [mem 0xffff8d743cc54000-0xffff8d743ffbffff] [ 2.587366] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.589993] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.593304] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.122513] Initialise system trusted keyrings [ 3.124889] Key type blacklist registered [ 3.133621] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.146700] zbud: loaded [ 3.152825] *** VALIDATE nfs *** [ 3.154855] *** VALIDATE nfs4 *** [ 3.157694] pstore: using deflate compression [ 3.162713] Platform Keyring initialized [ 3.277622] NET: Registered protocol family 38 [ 3.279953] Key type asymmetric registered [ 3.281701] Asymmetric key parser 'x509' registered [ 3.283837] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.287216] io scheduler mq-deadline registered [ 3.289305] io scheduler kyber registered [ 3.291231] io scheduler bfq registered [ 3.293247] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.296668] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.299626] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.302724] ACPI: Power Button [PWRF] [ 3.308750] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.316169] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.330412] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.337815] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.364224] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.392092] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.420568] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.426744] Non-volatile memory driver v1.3 [ 3.428553] Linux agpgart interface v0.103 [ 3.466820] virtio_blk virtio1: [vda] 146960 512-byte logical blocks (75.2 MB/71.8 MiB) [ 3.471900] vda: detected capacity change from 0 to 75243520 [ 3.495041] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.498646] vdb: detected capacity change from 0 to 1073741824 [ 3.522415] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.527642] vdc: detected capacity change from 0 to 2621440000 [ 3.548356] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.551469] vdd: detected capacity change from 0 to 2621440000 [ 3.574099] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.576960] vde: detected capacity change from 0 to 4294967296 [ 3.592859] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.595832] vdf: detected capacity change from 0 to 4294967296 [ 3.602819] libphy: Fixed MDIO Bus: probed [ 3.608888] usbcore: registered new interface driver usbserial_generic [ 3.611755] usbserial: USB Serial support registered for generic [ 3.614226] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.618826] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.620842] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.623571] mousedev: PS/2 mouse device common for all mice [ 3.628918] rtc_cmos 00:05: RTC can wake from S4 [ 3.632143] rtc_cmos 00:05: registered as rtc0 [ 3.633851] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.635200] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.641806] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.642493] intel_pstate: CPU model not supported [ 3.647359] hid: raw HID events driver (C) Jiri Kosina [ 3.650681] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.651634] usbcore: registered new interface driver usbhid [ 3.657488] usbhid: USB HID core driver [ 3.659954] drop_monitor: Initializing network drop monitor service [ 3.662450] Initializing XFRM netlink socket [ 3.664802] NET: Registered protocol family 10 [ 3.668205] Segment Routing with IPv6 [ 3.670323] NET: Registered protocol family 17 [ 3.672779] mpls_gso: MPLS GSO support [ 3.678466] RAS: Correctable Errors collector initialized. [ 3.680835] AVX version of gcm_enc/dec engaged. [ 3.682493] AES CTR mode by8 optimization enabled [ 3.767867] sched_clock: Marking stable (3767822146, 0)->(4823830687, -1056008541) [ 3.771642] registered taskstats version 1 [ 3.773867] Loading compiled-in X.509 certificates [ 3.775886] zswap: loaded using pool lzo/zbud [ 3.806108] Key type big_key registered [ 3.821489] Key type encrypted registered [ 3.823568] ima: No TPM chip found, activating TPM-bypass! [ 3.826172] ima: Allocated hash algorithm: sha1 [ 3.827925] ima: No architecture policies found [ 3.830089] evm: Initialising EVM extended attributes: [ 3.832260] evm: security.selinux [ 3.833819] evm: security.ima [ 3.835103] evm: security.capability [ 3.836674] evm: HMAC attrs: 0x1 [ 3.839559] rtc_cmos 00:05: setting system clock to 2026-08-27 12:42:52 UTC (1787834572) [ 3.846672] debug: unmapping init [mem 0xffffffff9f203000-0xffffffff9f3fffff] [ 3.850191] debug: unmapping init [mem 0xffffffff9df82000-0xffffffff9e258fff] [ 3.855193] Write protecting the kernel read-only data: 28672k [ 3.858868] debug: unmapping init [mem 0xffffffff9c603000-0xffffffff9c7fffff] [ 3.862128] debug: unmapping init [mem 0xffffffff9cf14000-0xffffffff9cffffff] [ 3.915119] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.922166] systemd[1]: Detected virtualization kvm. [ 3.923577] systemd[1]: Detected architecture x86-64. [ 3.925169] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.951851] systemd[1]: No hostname configured. [ 3.953245] systemd[1]: Set hostname to . [ 3.954986] random: systemd: uninitialized urandom read (16 bytes read) [ 3.957205] systemd[1]: Initializing machine ID from random generator. [ 4.010830] random: ln: uninitialized urandom read (6 bytes read) [ 4.097990] random: systemd: uninitialized urandom read (16 bytes read) [ 4.104194] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.115050] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 4.119310] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.859379] device-mapper: uevent: version 1.0.3 [ 4.861917] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.722166] random: fast init done [ 5.757570] virtio_net virtio0 ens2: renamed from eth0 [ 5.784694] scsi host0: ata_piix [ 5.811243] scsi host1: ata_piix [ 5.813840] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.816323] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.341979] random: crng init done [ 10.345743] random: 7 urandom warning(s) missed due to ratelimiting [ 10.360982] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.967452] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.180922] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.427399] SELinux: Disabled at runtime. [ 12.487794] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.498033] systemd[1]: Detected virtualization kvm. [ 12.500456] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.020176] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.022660] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.029801] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.034224] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.038550] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.045902] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.052116] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for det[ 13.128770] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ails. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.434920] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.732791] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.750034] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.887267] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.910336] EDAC sbridge: Ver: 1.1.2 [ 15.664119] Key type dns_resolver registered [ 15.995517] NFS: Registering the id_resolver key type [ 15.997818] Key type id_resolver registered [ 15.999949] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg417-server login: [ 49.231240] spl: loading out-of-tree module taints kernel. [ 55.949256] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 69.602678] Key type ._llcrypt registered [ 69.604613] Key type .llcrypt registered [ 69.742024] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 93.759964] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 95.282801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 95.315736] alg: No test for adler32 (adler32-zlib) [ 96.816317] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 97.953139] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 99.712221] Key type lgssc registered [ 101.684173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.260407] hrtimer: interrupt took 5396062 ns [ 113.415633] vdc: vdc1 vdc9 [ 124.944914] vde: vde1 vde9 [ 135.908741] vdf: vdf1 vdf9 [ 155.971796] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 165.916892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 167.246516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 167.540691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.679381] Lustre: lustre-MDT0000: new disk, initializing [ 168.123908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.180090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.874837] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.802485] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 184.587399] Lustre: lustre-OST0000: new disk, initializing [ 184.594038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 184.598891] Lustre: Skipped 1 previous similar message [ 184.717648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.991956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 187.000898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 187.168228] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 190.919378] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.835828] Lustre: lustre-OST0001: new disk, initializing [ 201.843446] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 201.953318] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.237609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 210.246573] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 210.465479] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 210.497225] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.736574] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.216455] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 237.663365] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing check_logdir /tmp/testlogs/ [ 243.509358] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing yml_node [ 248.650776] Lustre: DEBUG MARKER: Client: 2.17.57.98 [ 251.761904] Lustre: DEBUG MARKER: MDS: 2.17.57.98 [ 254.940914] Lustre: DEBUG MARKER: OSS: 2.17.57.98 [ 256.980596] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-quota ============----- Thu Aug 27 08:47:02 EDT 2026 [ 276.022768] Lustre: DEBUG MARKER: excepting tests: 2 4a 63 65 [ 277.962208] Lustre: DEBUG MARKER: skipping tests SLOW=no: 61 12a 9 [ 279.983937] Lustre: DEBUG MARKER: === sanity-quota: start setup 08:47:26 (1787834846) === [ 285.781268] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing check_config_client /mnt/lustre [ 305.688895] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 310.388293] Lustre: 11353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 315.773378] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 319.946478] Lustre: DEBUG MARKER: === sanity-quota: finish setup 08:48:05 (1787834885) === [ 387.464270] Lustre: DEBUG MARKER: == sanity-quota test 0: Test basic quota performance ===== 08:49:13 (1787834953) [ 448.072530] Lustre: DEBUG MARKER: == sanity-quota test 1a: Block hard limit (normal use and out of quota) ========================================================== 08:50:13 (1787835013) [ 465.629443] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 472.900604] Lustre: DEBUG MARKER: Write... [ 475.643063] Lustre: DEBUG MARKER: Write out of block quota ... [ 523.336750] Lustre: DEBUG MARKER: -------------------------------------- [ 525.182611] Lustre: DEBUG MARKER: Group quota (block hardlimit:10 MB) [ 531.815429] Lustre: DEBUG MARKER: Write... [ 534.251422] Lustre: DEBUG MARKER: Write out of block quota ... [ 580.427484] Lustre: DEBUG MARKER: -------------------------------------- [ 582.326125] Lustre: DEBUG MARKER: Project quota (block hardlimit:10 mb) [ 586.092329] Lustre: DEBUG MARKER: Write... [ 589.044400] Lustre: DEBUG MARKER: Write out of block quota ... [ 659.123734] Lustre: DEBUG MARKER: == sanity-quota test 1b: Quota pools: Block hard limit (normal use and out of quota) ========================================================== 08:53:44 (1787835224) [ 676.913514] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 695.804108] Lustre: DEBUG MARKER: Write... [ 698.839460] Lustre: DEBUG MARKER: Write out of block quota ... [ 745.223606] Lustre: DEBUG MARKER: -------------------------------------- [ 746.943917] Lustre: DEBUG MARKER: Group quota (block hardlimit:20 MB) [ 754.845663] Lustre: DEBUG MARKER: Write... [ 757.696034] Lustre: DEBUG MARKER: Write out of block quota ... [ 809.715743] Lustre: DEBUG MARKER: -------------------------------------- [ 811.870090] Lustre: DEBUG MARKER: Project quota (block hardlimit:20 mb) [ 815.624967] Lustre: DEBUG MARKER: Write... [ 818.977977] Lustre: DEBUG MARKER: Write out of block quota ... [ 893.431617] Lustre: DEBUG MARKER: == sanity-quota test 1c: Quota pools: check 3 pools with hardlimit only for global ========================================================== 08:57:39 (1787835459) [ 909.759925] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 932.319727] Lustre: DEBUG MARKER: Write... [ 935.713537] Lustre: DEBUG MARKER: Write out of block quota ... [ 1037.751585] Lustre: DEBUG MARKER: == sanity-quota test 1d: Quota pools: check block hardlimit on different pools ========================================================== 09:00:03 (1787835603) [ 1055.712228] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 1077.289670] Lustre: DEBUG MARKER: Write... [ 1079.752837] Lustre: DEBUG MARKER: Write out of block quota ... [ 1173.770390] Lustre: DEBUG MARKER: == sanity-quota test 1e: Quota pools: global pool high block limit vs quota pool with small ========================================================== 09:02:19 (1787835739) [ 1191.218751] Lustre: DEBUG MARKER: User quota (block hardlimit:53000000 MB) [ 1205.884948] Lustre: DEBUG MARKER: Write... [ 1209.854445] Lustre: DEBUG MARKER: Write out of block quota ... [ 1225.229890] Lustre: DEBUG MARKER: Write... [ 1292.730656] Lustre: DEBUG MARKER: == sanity-quota test 1f: Quota pools: correct qunit after removing/adding OST ========================================================== 09:04:18 (1787835858) [ 1308.979735] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 1321.610875] Lustre: DEBUG MARKER: Write... [ 1324.088514] Lustre: DEBUG MARKER: Write out of block quota ... [ 1370.771283] Lustre: DEBUG MARKER: Write... [ 1373.675912] Lustre: DEBUG MARKER: Write out of block quota ... [ 1430.692854] Lustre: DEBUG MARKER: == sanity-quota test 1g: Quota pools: Block hard limit with wide striping ========================================================== 09:06:36 (1787835996) [ 1448.291270] Lustre: DEBUG MARKER: User quota (block hardlimit:40 MB) [ 1462.653495] Lustre: DEBUG MARKER: Write... [ 1476.288891] Lustre: DEBUG MARKER: Write out of block quota ... [ 1567.559023] Lustre: DEBUG MARKER: == sanity-quota test 1h: Block hard limit test using fallocate ========================================================== 09:08:52 (1787836132) [ 1569.532769] Lustre: DEBUG MARKER: SKIP: sanity-quota test_1h need >= 2.13.57 and ldiskfs for fallocate [ 1572.656188] Lustre: DEBUG MARKER: == sanity-quota test 1i: Quota pools: different limit and usage relations ========================================================== 09:08:57 (1787836137) [ 1591.445488] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 1605.088907] Lustre: DEBUG MARKER: Write... [ 1608.024454] Lustre: DEBUG MARKER: Write out of block quota ... [ 1648.623518] Lustre: DEBUG MARKER: Write... [ 1652.287512] Lustre: DEBUG MARKER: Write out of block quota ... [ 1664.597321] LustreError: 12493:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:15366 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 1714.954868] Lustre: DEBUG MARKER: == sanity-quota test 1j: Enable project quota enforcement for root ========================================================== 09:11:20 (1787836280) [ 1739.246848] Lustre: DEBUG MARKER: -------------------------------------- [ 1742.365989] Lustre: DEBUG MARKER: Project quota (hardlimit: 40 mb, 2048 files) [ 2163.590684] Lustre: DEBUG MARKER: == sanity-quota test 1k: LQA: Block hard limit (normal use and out of quota) ========================================================== 09:18:49 (1787836729) [ 2187.687719] Lustre: DEBUG MARKER: Write... [ 2193.093145] Lustre: DEBUG MARKER: Write out of block quota ... [ 2236.774916] Lustre: DEBUG MARKER: Write... [ 2240.736595] Lustre: DEBUG MARKER: Write out of block quota ... [ 2282.535461] Lustre: DEBUG MARKER: Write... [ 2286.814528] Lustre: DEBUG MARKER: Write out of block quota ... [ 2333.732873] Lustre: DEBUG MARKER: == sanity-quota test 1l: Async writes should not be rejected by quota with root_prj_enable ========================================================== 09:21:39 (1787836899) [ 2400.667163] Lustre: DEBUG MARKER: SKIP: sanity-quota test_2 skipping excluded test 2 [ 2402.361839] Lustre: DEBUG MARKER: == sanity-quota test 3a: Block soft limit (start timer, timer goes off, stop timer) ========================================================== 09:22:48 (1787836968) [ 2494.302818] Lustre: DEBUG MARKER: Write after timer goes off [ 2496.259212] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2637.554553] Lustre: DEBUG MARKER: Write after timer goes off [ 2639.373149] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2781.417438] Lustre: DEBUG MARKER: Write after timer goes off [ 2783.614555] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2901.194299] Lustre: DEBUG MARKER: == sanity-quota test 3b: Quota pools: Block soft limit (start timer, expires, stop timer) ========================================================== 09:31:06 (1787837466) [ 3000.710695] Lustre: DEBUG MARKER: Write after timer goes off [ 3003.071327] Lustre: DEBUG MARKER: Write after cancel lru locks [ 3155.040865] Lustre: DEBUG MARKER: Write after timer goes off [ 3157.062187] Lustre: DEBUG MARKER: Write after cancel lru locks [ 3304.272930] Lustre: DEBUG MARKER: Write after timer goes off [ 3307.367398] Lustre: DEBUG MARKER: Write after cancel lru locks [ 3438.891710] Lustre: DEBUG MARKER: == sanity-quota test 3c: Quota pools: check block soft limit on different pools ========================================================== 09:40:04 (1787838004) [ 3552.043825] Lustre: DEBUG MARKER: Write after timer goes off [ 3554.050570] Lustre: DEBUG MARKER: Write after cancel lru locks [ 3659.923364] Lustre: DEBUG MARKER: SKIP: sanity-quota test_4a skipping excluded test 4a [ 3662.110421] Lustre: DEBUG MARKER: == sanity-quota test 4b: Grace time strings handling ===== 09:43:47 (1787838227) [ 3680.779193] Lustre: DEBUG MARKER: == sanity-quota test 5: Chown [ 3833.612577] Lustre: DEBUG MARKER: == sanity-quota test 6: Test dropping acquire request on master ========================================================== 09:46:39 (1787838399) [ 3883.489989] Lustre: *** cfs_fail_loc=513, val=601*** [ 3884.564932] Lustre: *** cfs_fail_loc=513, val=601*** [ 3884.572776] Lustre: Skipped 3 previous similar messages [ 3885.331343] LustreError: 14413:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1874680524536320 [ 3886.049514] Lustre: *** cfs_fail_loc=513, val=601*** [ 3886.051551] Lustre: Skipped 20 previous similar messages [ 3888.609784] Lustre: *** cfs_fail_loc=513, val=601*** [ 3888.617026] Lustre: Skipped 1 previous similar message [ 3893.728893] Lustre: *** cfs_fail_loc=513, val=601*** [ 3893.737791] Lustre: Skipped 10 previous similar messages [ 3900.898185] Lustre: 3301:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838453/real 1787838453] req@ffff8d7387086d00 x1874680524536320/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787838469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 3900.956612] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.000683] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 3901.012407] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3901.922055] Lustre: *** cfs_fail_loc=513, val=601*** [ 3901.930368] Lustre: Skipped 21 previous similar messages [ 3903.109376] LustreError: 14406:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1874680524540928 [ 3916.772359] LustreError: 14407:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1874680524544128 [ 3916.794935] LustreError: 14407:0:(service.c:2341:ptlrpc_server_handle_req_in()) Skipped 2 previous similar messages [ 3919.328224] Lustre: 38797:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838471/real 1787838471] req@ffff8d7387086d00 x1874680524540928/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787838487 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_005.0' uid:0 gid:0 projid:4294967295 [ 3919.332251] Lustre: *** cfs_fail_loc=513, val=601*** [ 3919.370334] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.374642] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 3919.374988] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3919.427323] Lustre: Skipped 43 previous similar messages [ 3922.450988] LustreError: 14413:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1874680524545536 [ 3932.642269] Lustre: 3303:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838485/real 1787838485] req@ffff8d7499739a40 x1874680524544000/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787838501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 3932.642947] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.672540] Lustre: 3303:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3932.700607] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0001_UUID (at 0@lo) reconnecting [ 3932.714857] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3938.785230] Lustre: 16127:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838491/real 1787838491] req@ffff8d7382ef7480 x1874680524545536/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787838507 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_003.0' uid:0 gid:0 projid:4294967295 [ 3938.803873] Lustre: 16127:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3938.807931] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.827995] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 3938.838631] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3983.051819] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 09:49:08 (1787838548) [ 4018.211529] Lustre: Failing over lustre-OST0000 [ 4018.662281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.719539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4020.565886] Lustre: server umount lustre-OST0000 complete [ 4027.124465] LustreError: 6684:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4027.145633] LustreError: 6684:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4028.906129] LustreError: 38742:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4032.234880] LustreError: 7989:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.461051] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4035.504696] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4036.735599] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4036.928167] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4036.931766] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4042.188374] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4052.283766] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4060.188985] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4066.692064] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4073.403815] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4080.044988] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4086.585345] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4122.540935] Lustre: Failing over lustre-OST0000 [ 4122.599947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4122.613805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4122.706861] Lustre: server umount lustre-OST0000 complete [ 4127.719190] LustreError: 39119:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4127.750048] LustreError: 39119:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4133.092352] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4133.122588] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4134.500923] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4134.602893] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4134.608507] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4140.228485] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4149.003134] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4155.883426] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4164.100155] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4170.823922] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4177.821247] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4184.523956] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4222.744531] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) ========================================================== 09:53:08 (1787838788) [ 4267.621159] Lustre: *** cfs_fail_loc=a02, val=0*** [ 4275.743803] LustreError: 3303:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8 qsd:lustre-OST0000 qtype:usr lqe: ffff8d74b63fc600 id:60000 enforced:1 granted: 1026 pending:0 waiting:0 req:1 usage: 2052 qunit:0 qtune:0 edquot:0 default:no revoke:0 [ 4275.766925] Lustre: Failing over lustre-OST0000 [ 4275.830900] Lustre: server umount lustre-OST0000 complete [ 4276.719369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4276.736179] LustreError: 38742:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4285.455953] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4285.494457] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4287.318066] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4287.592648] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4287.593557] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4292.967824] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4303.068727] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4309.009663] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4314.942991] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4320.534477] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4326.934176] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4333.068815] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4388.104431] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ========================================================== 09:55:53 (1787838953) [ 4416.678495] LustreError: 90339:0:(qsd_reint.c:488:qsd_reint_main()) cfs_fail_timeout id a03 sleeping for 10000ms [ 4418.981158] Lustre: Failing over lustre-MDT0000 [ 4419.471168] Lustre: server umount lustre-MDT0000 complete [ 4423.904295] LustreError: 90340:0:(qsd_reint.c:488:qsd_reint_main()) cfs_fail_timeout interrupted [ 4430.517431] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4430.761161] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.975324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4431.077836] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4431.656397] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4431.727720] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4431.775590] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:132 to 0x280000400:161) [ 4431.775642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:129 to 0x240000400:161) [ 4435.940620] LustreError: 3299:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x2020000:0x0].0x0 (ffff8d7483d06d00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4435.977459] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4436.019041] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4440.032450] Lustre: 3303:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838992/real 1787838992] req@ffff8d7385b361c0 x1874680524973952/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787839008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4440.063620] Lustre: 3303:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4445.412977] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4538.654481] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4545.044650] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4551.594641] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4557.987471] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4563.209268] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4603.299633] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ========================================================== 09:59:28 (1787839168) [ 4632.038194] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4638.050535] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4692.762587] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================================== 10:00:58 (1787839258) [ 4694.861942] Lustre: DEBUG MARKER: SKIP: sanity-quota test_7e needs >= 2 MDTs [ 4696.720966] Lustre: DEBUG MARKER: == sanity-quota test 7f: Quota reintegration automatically ========================================================== 10:01:02 (1787839262) [ 4715.814099] Lustre: *** cfs_fail_loc=a11, val=0*** [ 4715.815923] Lustre: Skipped 2 previous similar messages [ 4722.433801] Lustre: *** cfs_fail_loc=a11, val=0*** [ 4776.931705] Lustre: *** cfs_fail_loc=a11, val=0*** [ 4930.612904] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ==== 10:04:56 (1787839496) [ 5132.756722] Lustre: DEBUG MARKER: SKIP: sanity-quota test_9 skipping SLOW test 9 [ 5135.112995] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ======== 10:08:20 (1787839700) [ 5192.514954] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota ======= 10:09:18 (1787839758) [ 5249.256413] Lustre: DEBUG MARKER: SKIP: sanity-quota test_12a skipping SLOW test 12a [ 5251.240301] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ======== 10:10:16 (1787839816) [ 5253.039383] Lustre: DEBUG MARKER: SKIP: sanity-quota test_12b needs >= 2 MDTs [ 5255.149674] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================================== 10:10:20 (1787839820) [ 5328.201505] Lustre: DEBUG MARKER: == sanity-quota test 14: check panic in qmt_site_recalc_cb ========================================================== 10:11:34 (1787839894) [ 5354.883890] Lustre: Failing over lustre-OST0000 [ 5355.045628] Lustre: server umount lustre-OST0000 complete [ 5357.545437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5357.559916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5357.590330] Lustre: Skipped 1 previous similar message [ 5357.599713] LustreError: 100671:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5357.628039] LustreError: 100671:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5362.658402] LustreError: 100685:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5362.688329] LustreError: 100685:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5367.792986] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5367.813732] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5368.551342] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5369.316093] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5369.331302] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5369.354614] Lustre: Skipped 1 previous similar message [ 5374.331837] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5417.791277] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ========= 10:13:03 (1787839983) [ 5460.142969] Lustre: DEBUG MARKER: == sanity-quota test 16a: lfs quota should skip the inactive MDT/OST ========================================================== 10:13:44 (1787840024) [ 5491.853589] Lustre: lustre-OST0000: Client 4d1fbaee-e469-4476-bda4-3312328fdec3 (at 192.168.204.17@tcp) reconnecting [ 5518.521576] Lustre: DEBUG MARKER: == sanity-quota test 16b: lfs quota should skip the nonexistent MDT/OST ========================================================== 10:14:44 (1787840084) [ 5520.732503] Lustre: DEBUG MARKER: SKIP: sanity-quota test_16b needs >= 3 MDTs [ 5523.293863] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error == 10:14:48 (1787840088) [ 5552.944506] Lustre: *** cfs_fail_loc=a04, val=37*** [ 5552.949843] LustreError: 16127:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x1 qsd:lustre-OST0001 qtype:usr lqe: ffff8d74bb507080 id:60000 enforced:1 granted: 0 pending:0 waiting:1 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 [ 5554.046587] Lustre: *** cfs_fail_loc=a04, val=37*** [ 5554.051406] LustreError: 6690:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x1 qsd:lustre-OST0001 qtype:usr lqe: ffff8d74bb507080 id:60000 enforced:1 granted: 0 pending:0 waiting:1024 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 [ 5556.173356] Lustre: *** cfs_fail_loc=a04, val=37*** [ 5556.187767] LustreError: 6691:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x1 qsd:lustre-OST0001 qtype:usr lqe: ffff8d74bb507080 id:60000 enforced:1 granted: 0 pending:0 waiting:1024 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 [ 5627.758303] Lustre: *** cfs_fail_loc=a04, val=11*** [ 5627.765367] Lustre: Skipped 1 previous similar message [ 5698.806225] Lustre: *** cfs_fail_loc=a04, val=110*** [ 5698.809901] Lustre: Skipped 2 previous similar messages [ 5768.847102] Lustre: *** cfs_fail_loc=a04, val=107*** [ 5768.853957] Lustre: Skipped 2 previous similar messages [ 5877.169638] Lustre: DEBUG MARKER: == sanity-quota test 18: MDS failover while writing, no watchdog triggered (b14840) ========================================================== 10:20:42 (1787840442) [ 5900.669960] Lustre: DEBUG MARKER: User quota (limit: 200) [ 5908.099184] Lustre: DEBUG MARKER: Write 100M (buffered) ... [ 5912.182945] LustreError: 120144:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 5913.253131] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5915.408825] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 5917.580190] Lustre: Failing over lustre-MDT0000 [ 5918.019593] Lustre: server umount lustre-MDT0000 complete [ 5936.099260] Lustre: 3302:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840488/real 1787840488] req@ffff8d74c1cd8b40 x1874680526137216/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787840504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 5936.137611] Lustre: 3302:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5936.148578] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5937.164576] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5937.176922] Lustre: 3303:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840489/real 1787840489] req@ffff8d74b80cde00 x1874680526137344/t0(0) o400->MGC192.168.204.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787840505 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5937.202912] Lustre: 3303:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5937.713783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5937.786178] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5938.831349] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5939.003048] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5939.040751] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:559 to 0x280000400:577) [ 5939.040959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:564 to 0x240000400:609) [ 5941.601162] Lustre: 3303:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840494/real 1787840494] req@ffff8d74bb1183c0 x1874680526137856/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787840510 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5942.174940] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5945.762987] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5946.784946] Lustre: 3303:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840499/real 1787840499] req@ffff8d7382f51e00 x1874680526138240/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787840515 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5946.808325] Lustre: 3303:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5952.995528] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5955.169443] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5963.746799] Lustre: DEBUG MARKER: (dd_pid=113757, time=3, timeout=600) [ 6014.316412] Lustre: DEBUG MARKER: User quota (limit: 200) [ 6020.970573] Lustre: DEBUG MARKER: Write 100M (directio) ... [ 6024.409416] LustreError: 123006:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 6025.174971] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6026.947339] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 6029.208369] Lustre: Failing over lustre-MDT0000 [ 6029.563103] Lustre: server umount lustre-MDT0000 complete [ 6047.201046] Lustre: 3301:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840599/real 1787840599] req@ffff8d7381ba92c0 x1874680526178560/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787840615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 6047.242373] Lustre: 3301:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6047.252914] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6047.262825] Lustre: Skipped 1 previous similar message [ 6049.470739] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6049.909480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6049.979498] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6052.902860] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6052.914923] Lustre: Skipped 1 previous similar message [ 6054.282565] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6056.677428] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 6056.832084] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 6056.865738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641) [ 6056.865882] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:559 to 0x280000400:609) [ 6063.638327] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6065.245082] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6075.595506] Lustre: DEBUG MARKER: (dd_pid=116168, time=4, timeout=600) [ 6142.415389] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 10:25:08 (1787840708) [ 6168.668265] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 6170.582046] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 6178.092712] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 6180.648771] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 6217.900976] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 10:26:23 (1787840783) [ 6253.370904] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [ 6270.136402] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [ 6272.319401] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [ 6274.395604] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000 [ 6276.848613] Lustre: DEBUG MARKER: Set quota for 1 times [ 6281.508623] Lustre: DEBUG MARKER: Set quota for 2 times [ 6285.345765] Lustre: DEBUG MARKER: Set quota for 3 times [ 6289.554527] Lustre: DEBUG MARKER: Set quota for 4 times [ 6293.710400] Lustre: DEBUG MARKER: Set quota for 5 times [ 6298.465280] Lustre: DEBUG MARKER: Set quota for 6 times [ 6302.753449] Lustre: DEBUG MARKER: Set quota for 7 times [ 6349.161626] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 10:28:34 (1787840914) [ 6363.177712] Lustre: server umount lustre-MDT0000 complete [ 6368.893862] LustreError: 14291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840937 with bad export cookie 4273330425824043999 [ 6368.911061] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6378.718483] Lustre: server umount lustre-OST0000 complete [ 6379.425954] Lustre: 3301:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840932/real 1787840932] req@ffff8d7381864780 x1874680526295168/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787840948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6379.467449] Lustre: 3301:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 6379.487599] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6379.500923] Lustre: Skipped 1 previous similar message [ 6383.874672] Lustre: server umount lustre-OST0001 complete [ 6402.607729] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 6414.736226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6420.385862] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6423.905908] Lustre: 132709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6430.900709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6438.718488] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6440.112045] LustreError: 133088:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6440.147597] LustreError: 133088:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6440.156864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:644 to 0x240000400:673) [ 6445.537465] LustreError: 133088:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6447.467710] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6452.727981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:611 to 0x280000400:641) [ 6454.561645] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6462.741056] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6466.560869] Lustre: 134636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6485.739881] Lustre: server umount lustre-MDT0000 complete [ 6492.033685] LustreError: 132253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841060 with bad export cookie 4273330425824050810 [ 6492.051480] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6501.446670] Lustre: server umount lustre-OST0000 complete [ 6504.418191] Lustre: 3302:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841057/real 1787841057] req@ffff8d74af0ee580 x1874680526338176/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787841073 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6504.460102] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6506.264520] Lustre: server umount lustre-OST0001 complete [ 6525.943682] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 6539.246851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6544.680887] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6547.864802] Lustre: 137290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6562.203601] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6563.373183] LustreError: 137658:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6563.397453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:644 to 0x240000400:705) [ 6568.423985] LustreError: 137658:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6571.341547] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6571.347499] Lustre: Skipped 1 previous similar message [ 6576.656624] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:611 to 0x280000400:673) [ 6578.438190] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6588.114667] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6593.239721] Lustre: 139206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6608.866795] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 10:32:54 (1787841174) [ 6610.938662] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS [ 6613.086559] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 10:32:58 (1787841178) [ 6679.555536] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 10:34:05 (1787841245) [ 6737.580818] Lustre: DEBUG MARKER: Write... [ 6740.205319] Lustre: DEBUG MARKER: Write out of block quota ... [ 6804.005257] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 10:36:09 (1787841369) [ 6812.361209] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 10:36:18 (1787841378) [ 6823.544797] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 10:36:29 (1787841389) [ 6834.384403] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 10:36:39 (1787841399) [ 6843.970221] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 10:36:49 (1787841409) [ 6915.802229] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user [ 7090.833905] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user [ 7285.120859] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 10:44:10 (1787841850) [ 7351.814223] Lustre: DEBUG MARKER: Restart... [ 7359.970873] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7359.993479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7362.437126] Lustre: server umount lustre-MDT0000 complete [ 7367.240021] LustreError: 136843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841935 with bad export cookie 4273330425824052343 [ 7367.255698] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7367.445671] Lustre: server umount lustre-OST0000 complete [ 7372.257665] Lustre: server umount lustre-OST0001 complete [ 7388.259076] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 7399.424319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.557831] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7407.375374] Lustre: 157745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7413.855369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7421.452668] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7424.034848] LustreError: 158106:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7424.046793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:714 to 0x240000400:737) [ 7429.092142] LustreError: 158594:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7431.176320] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7436.294113] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:682 to 0x280000400:705) [ 7439.634294] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7448.508308] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7452.758728] Lustre: 159658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7523.221333] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 10:48:09 (1787842089) [ 7597.486915] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 10:49:23 (1787842163) [ 9524.921694] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 11:21:30 (1787844090) [ 9543.341384] Lustre: server umount lustre-MDT0000 complete [ 9548.387318] LustreError: 157299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844117 with bad export cookie 4273330425824058839 [ 9548.401912] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9559.160145] Lustre: server umount lustre-OST0000 complete [ 9559.841111] Lustre: 3303:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844112/real 1787844112] req@ffff8d73d06ab0c0 x1874680529611520/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787844128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9559.886535] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9559.897809] Lustre: Skipped 1 previous similar message [ 9563.351212] Lustre: server umount lustre-OST0001 complete [ 9579.199419] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 9590.302579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9594.678439] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9597.799891] Lustre: 168098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9603.852284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9609.579335] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9615.075877] LustreError: 168470:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9615.110083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5740 to 0x240000400:5761) [ 9618.361567] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9623.561867] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5706 to 0x280000400:5729) [ 9624.617975] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9632.293907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9635.875566] Lustre: 170016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9672.084648] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 11:23:57 (1787844237) [ 9717.644736] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 11:24:43 (1787844283) [ 9764.894673] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 11:25:30 (1787844330) [ 9766.379053] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs [ 9768.498975] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 11:25:34 (1787844334) [ 9770.222965] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs [ 9772.144662] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 11:25:37 (1787844337) [ 9869.522324] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 11:27:15 (1787844435) [ 9923.926061] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 11:28:09 (1787844489) [10021.925945] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 11:29:47 (1787844587) [10034.527553] Lustre: *** cfs_fail_loc=a09, val=0*** [10034.529708] Lustre: Skipped 2 previous similar messages [10036.555982] Lustre: *** cfs_fail_loc=a09, val=0*** [10036.558042] Lustre: Skipped 97 previous similar messages [10040.577458] Lustre: *** cfs_fail_loc=a09, val=0*** [10040.580125] Lustre: Skipped 181 previous similar messages [10048.588908] Lustre: *** cfs_fail_loc=a09, val=0*** [10048.591973] Lustre: Skipped 291 previous similar messages [10064.626107] Lustre: *** cfs_fail_loc=a09, val=0*** [10064.631971] Lustre: Skipped 693 previous similar messages [10096.633408] Lustre: *** cfs_fail_loc=a09, val=0*** [10096.636491] Lustre: Skipped 1205 previous similar messages [10160.647966] Lustre: *** cfs_fail_loc=a09, val=0*** [10160.652570] Lustre: Skipped 2479 previous similar messages [10567.460309] Lustre: *** cfs_fail_loc=a08, val=0*** [10567.465758] Lustre: Skipped 3050 previous similar messages [10567.468129] Lustre: *** cfs_fail_loc=a08, val=0*** [10567.474634] Lustre: Skipped 1 previous similar message [10877.745399] LustreError: lustre-MDT0000-lwp-OST0001: operation quota_acquire to node 0@lo failed: rc = -107 [10877.749164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10877.751872] LustreError: Skipped 1 previous similar message [10877.754436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10877.754442] Lustre: Skipped 1 previous similar message [10877.796478] Lustre: Skipped 1 previous similar message [10881.939640] Lustre: server umount lustre-MDT0000 complete [10887.244665] LustreError: 3303:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8d73c3b121c0 x1874680534741248/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 [10887.271683] LustreError: 3303:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x4 qsd:lustre-OST0000 qtype:grp lqe: ffff8d74856d1c80 id:274 enforced:1 granted: 16384 pending:0 waiting:0 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 [10887.295520] LustreError: 3303:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message [10887.322641] LustreError: 167650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845455 with bad export cookie 4273330425825839800 [10887.366186] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10887.721938] Lustre: server umount lustre-OST0000 complete [10892.825457] Lustre: server umount lustre-OST0001 complete [10901.010290] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [10909.675715] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [10921.585772] vdc: vdc1 vdc9 [10932.702739] vde: vde1 vde9 [10932.725671] vde: vde1 vde9 [10944.527688] vdf: vdf1 vdf9 [10968.307871] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [10979.895860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10980.102209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10980.156533] Lustre: lustre-MDT0000: new disk, initializing [10980.517087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10980.570706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10985.976743] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10990.954101] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10997.761486] Lustre: lustre-OST0000: new disk, initializing [10997.764908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10997.768976] Lustre: Skipped 1 previous similar message [10997.847243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10999.684754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10999.696685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10999.940432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11004.679211] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11015.733216] Lustre: lustre-OST0001: new disk, initializing [11015.738609] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11015.847834] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11017.585632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [11017.598367] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [11017.719305] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [11022.710949] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11033.499132] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11037.280499] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11079.133795] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 11:47:24 (1787845644) [11088.211081] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 11:47:33 (1787845653) [11144.225701] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 11:48:29 (1787845709) [11209.595874] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 11:49:34 (1787845774) [11245.000784] Lustre: DEBUG MARKER: rename directory return 255 [11297.145583] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 11:51:02 (1787845862) [11331.819859] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 11:51:37 (1787845897) [11380.257439] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 11:52:25 (1787845945) [11520.904367] LustreError: 194864:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [11560.526960] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 11:55:25 (1787846125) [11602.603709] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 11:56:08 (1787846168) [11650.645446] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 11:56:55 (1787846215) [11861.912459] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 12:00:26 (1787846426) [11864.120740] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test [11866.482624] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 12:00:32 (1787846432) [11935.985605] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61 [11938.192473] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 12:01:43 (1787846503) [11973.458405] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63 [11975.817412] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 12:02:21 (1787846541) [12035.736598] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65 [12037.489436] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 12:03:23 (1787846603) [12088.773494] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 12:04:14 (1787846654) [12090.727346] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode [12092.594602] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 12:04:18 (1787846658) [12124.970063] LustreError: 207778:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1 [12188.339165] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 12:05:53 (1787846753) [12220.011481] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [12222.453276] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [12437.676342] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 12:10:03 (1787847003) [12495.568646] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 12:11:00 (1787847060) [12535.824409] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 12:11:41 (1787847101) [12537.919375] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode [12540.114647] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 12:11:45 (1787847105) [12541.895938] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode [12543.928422] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 12:11:49 (1787847109) [12563.371588] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB) [12584.095767] Lustre: DEBUG MARKER: Write... [12588.884605] Lustre: DEBUG MARKER: Write out of block quota ... [12652.154639] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 12:13:37 (1787847217) [12680.723670] Lustre: DEBUG MARKER: set to use default quota [12682.610150] Lustre: DEBUG MARKER: set default quota [12684.371840] Lustre: DEBUG MARKER: get default quota [12691.817310] Lustre: DEBUG MARKER: Test not out of quota [12697.097854] Lustre: DEBUG MARKER: Test out of quota [12711.001228] Lustre: DEBUG MARKER: Increase default quota [12746.998733] Lustre: DEBUG MARKER: Set quota to override default quota [12747.042850] LustreError: 184806:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1788452115 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [12760.809289] Lustre: DEBUG MARKER: Set to use default quota again [12784.654963] Lustre: DEBUG MARKER: Cleanup [12858.458204] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 12:17:04 (1787847424) [12887.015678] Lustre: DEBUG MARKER: set default quota for qpool1 [12888.790591] Lustre: DEBUG MARKER: Write from user that hasn't lqe [12941.844489] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 12:18:27 (1787847507) [13031.182431] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 12:19:56 (1787847596) [13105.767832] Lustre: DEBUG MARKER: Write... [13110.223675] Lustre: DEBUG MARKER: Write out of block quota ... [13242.166029] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 12:23:27 (1787847807) [13292.829754] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 12:24:18 (1787847858) [13301.087591] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 12:24:26 (1787847866) [13302.997354] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate [13305.524986] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 12:24:30 (1787847870) [13307.517846] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate [13309.759191] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 12:24:35 (1787847875) [13331.005779] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 12:24:56 (1787847896) [13332.831992] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode [13334.710165] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 12:25:00 (1787847900) [13352.523560] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [13359.281829] LustreError: 232625:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms [13359.291404] LustreError: 232625:0:(qmt_pool.c:1407:qmt_pool_recalc()) Skipped 5 previous similar messages [13362.147279] LustreError: 3302:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8d74bb845a40 x1874680535718016/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [13362.179094] LustreError: 3302:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 10 previous similar messages [13362.186207] LustreError: 3302:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8 qsd:lustre-MDT0000 qtype:grp lqe: ffff8d73c6d69c80 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 372 qunit:0 qtune:0 edquot:0 default:no revoke:0 [13362.198137] LustreError: 3302:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 4 previous similar messages [13362.658706] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13362.666924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13362.673062] Lustre: Skipped 1 previous similar message [13362.680189] Lustre: Skipped 1 previous similar message [13364.989834] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.17@tcp (stopping) [13367.778327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13367.784508] Lustre: Skipped 1 previous similar message [13369.345396] LustreError: 232625:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake [13369.749114] Lustre: server umount lustre-MDT0000 complete [13378.334482] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13378.935141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13379.055643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65) [13379.079115] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65) [13383.564401] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13392.364591] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13392.376121] Lustre: Skipped 1 previous similar message [13421.545993] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 12:26:26 (1787847986) [13457.203565] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 12:27:02 (1787848022) [13491.954485] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 12:27:37 (1787848057) [13549.392313] Lustre: *** cfs_fail_loc=a08, val=0*** [13549.396458] Lustre: Skipped 8 previous similar messages [13549.404138] Lustre: *** cfs_fail_loc=a08, val=0*** [13549.410625] Lustre: Skipped 15 previous similar messages [13674.462887] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 12:30:39 (1787848239) [13769.138944] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 12:32:14 (1787848334) [13855.345455] LustreError: 233208:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788453224 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [14023.838511] LustreError: 233209:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788453392 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [14194.197031] LustreError: 239265:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788453562 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [14324.685881] Lustre: DEBUG MARKER: == sanity-quota test 87: lfs quota -a should print default quota setting ========================================================== 12:41:30 (1787848890) [14337.061985] Lustre: *** cfs_fail_loc=a09, val=0*** [14341.090904] Lustre: *** cfs_fail_loc=a09, val=0*** [14341.095294] Lustre: Skipped 135 previous similar messages [14349.093714] Lustre: *** cfs_fail_loc=a09, val=0*** [14349.106151] Lustre: Skipped 284 previous similar messages [14438.889685] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14438.904113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14438.916768] Lustre: Skipped 1 previous similar message [14438.942345] Lustre: Skipped 1 previous similar message [14440.203757] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.17@tcp (stopping) [14442.115571] Lustre: server umount lustre-MDT0000 complete [14452.272293] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14452.908311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14453.102644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:68 to 0x240000400:97) [14453.109145] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:97) [14458.415569] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14458.856531] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14458.871476] Lustre: Skipped 1 previous similar message [14482.640675] Lustre: DEBUG MARKER: == sanity-quota test 88: Writing over quota should not hang ========================================================== 12:44:08 (1787849048) [14484.882164] Lustre: DEBUG MARKER: SKIP: sanity-quota test_88 require client with >4k pages [14487.299034] Lustre: DEBUG MARKER: == sanity-quota test 89: Show default quota with squash_uid ========================================================== 12:44:12 (1787849052) [14510.976627] Lustre: DEBUG MARKER: == sanity-quota test 90a: lfs quota should work without mount point ========================================================== 12:44:36 (1787849076) [14531.398787] Lustre: DEBUG MARKER: == sanity-quota test 90b: lfs quota should work with multiple mount points ========================================================== 12:44:57 (1787849097) [14553.781324] Lustre: DEBUG MARKER: == sanity-quota test 91: new quota index files in quota_master ========================================================== 12:45:19 (1787849119) [14571.489353] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14571.505094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14571.516915] Lustre: Skipped 1 previous similar message [14575.659577] Lustre: server umount lustre-MDT0000 complete [14579.572189] LustreError: 196379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849148 with bad export cookie 4273330425827342700 [14579.576739] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14579.587504] LustreError: 196379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14585.711819] Lustre: server umount lustre-OST0000 complete [14596.013895] Lustre: server umount lustre-OST0001 complete [14603.568942] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [14611.803603] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [14621.584641] vdc: vdc1 vdc9 [14631.898579] vde: vde1 vde9 [14645.070472] vdf: vdf1 vdf9 [14645.111626] vdf: vdf1 vdf9 [14645.141325] vdf: vdf1 vdf9 [14658.950742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14659.303418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14659.432629] Lustre: lustre-MDT0000: new disk, initializing [14660.054916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14660.217165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14666.341577] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14676.441702] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14684.821177] Lustre: lustre-OST0000: new disk, initializing [14684.825172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14684.831179] Lustre: Skipped 1 previous similar message [14684.928123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14686.324022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14686.349215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14686.600731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14692.890820] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14704.150520] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14712.214492] Lustre: lustre-OST0001: new disk, initializing [14712.222263] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [14712.342325] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14713.532658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [14713.549780] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [14713.692845] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [14720.443264] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14730.366279] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14744.058202] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14744.082778] Lustre: Skipped 1 previous similar message [14744.093474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14744.103983] Lustre: Skipped 1 previous similar message [14744.926326] Lustre: server umount lustre-MDT0000 complete [14758.946518] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14759.669155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14763.694313] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14763.703672] Lustre: Skipped 1 previous similar message [14765.999956] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14775.125514] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14776.860721] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14788.288665] Lustre: server umount lustre-MDT0000 complete [14793.291909] LustreError: 249979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849361 with bad export cookie 4273330425827345136 [14793.322601] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14803.075459] Lustre: server umount lustre-OST0000 complete [14805.985197] Lustre: 3302:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849358/real 1787849358] req@ffff8d7499738000 x1874680536643200/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [14806.040234] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14806.064063] Lustre: Skipped 1 previous similar message [14809.738239] LustreError: 3302:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8 qsd:lustre-OST0001 qtype:usr lqe: ffff8d74b035ed80 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2289 qunit:0 qtune:0 edquot:0 default:no revoke:0 [14809.976864] Lustre: server umount lustre-OST0001 complete [14830.002148] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [14838.258889] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [14849.189076] vdc: vdc1 vdc9 [14861.564841] vde: vde1 vde9 [14876.321095] vdf: vdf1 vdf9 [14876.340979] vdf: vdf1 vdf9 [14898.239788] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [14911.539733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14911.802592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14911.859899] Lustre: lustre-MDT0000: new disk, initializing [14912.542848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14912.589106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14918.088609] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14924.688106] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14934.652527] Lustre: lustre-OST0000: new disk, initializing [14934.668192] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14934.685545] Lustre: Skipped 1 previous similar message [14934.960985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14936.400837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14936.412269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14936.619265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14949.835315] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14970.722499] Lustre: lustre-OST0001: new disk, initializing [14970.733694] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [14970.944318] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14972.019368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [14972.025552] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [14972.197820] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [14979.409158] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14989.275630] Lustre: DEBUG MARKER: Using TIMEOUT=20 [14992.959581] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [15002.179624] Lustre: DEBUG MARKER: == sanity-quota test 92: Cannot set inode limit with Quota Pools ========================================================== 12:52:47 (1787849567) [15016.997878] Lustre: DEBUG MARKER: == sanity-quota test 93: update projid while client write to OST ========================================================== 12:53:02 (1787849582) [15046.829447] Lustre: *** cfs_fail_loc=170c, val=0*** [15114.737848] Lustre: DEBUG MARKER: == sanity-quota test 94: lfs quota all respects nodemap offset ========================================================== 12:54:40 (1787849680) [15161.323888] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15161.328303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15161.342276] Lustre: Skipped 1 previous similar message [15161.357429] Lustre: Skipped 1 previous similar message [15167.147669] Lustre: server umount lustre-MDT0000 complete [15177.424730] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15177.781681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15177.947121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [15183.093985] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15192.037854] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15192.056706] Lustre: Skipped 1 previous similar message [15194.396511] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.17@tcp (stopping) [15194.405308] Lustre: Skipped 3 previous similar messages [15197.164923] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15197.170910] Lustre: Skipped 1 previous similar message [15202.274682] LustreError: 265421:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15202.590212] Lustre: server umount lustre-MDT0000 complete [15216.026104] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15216.964852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [15221.995534] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15227.879789] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15227.885434] Lustre: Skipped 1 previous similar message [15230.861147] Lustre: DEBUG MARKER: == sanity-quota test 95a: Correct limits for squashed root ========================================================== 12:56:36 (1787849796) [15260.734108] Lustre: DEBUG MARKER: == sanity-quota test 95b: df respects squashed project id ========================================================== 12:57:06 (1787849826) [15315.365387] Lustre: DEBUG MARKER: == sanity-quota test 96: quota grant should be released when big files are deleted ========================================================== 12:58:01 (1787849881) [15317.365761] Lustre: DEBUG MARKER: SKIP: sanity-quota test_96 the test in only needed to run on LDiskFS [15319.567346] Lustre: DEBUG MARKER: == sanity-quota test 97a: LQA control commands =========== 12:58:05 (1787849885) [15349.377674] LustreError: 271016:0:(qmt_lqa.c:79:qmt_lqa_destroy()) lustre-QMT0000: cannot destroy lqa-dt-lqa1: rc = -2 [15349.388485] LustreError: 271016:0:(qmt_lqa.c:84:qmt_lqa_destroy()) lustre-QMT0000: cannot destroy lqa-md-lqa1: rc = -2 [15351.532566] Lustre: DEBUG MARKER: == sanity-quota test 97b: Check LQA internals ============ 12:58:37 (1787849917) [15371.592317] LustreError: 271811:0:(qmt_lqa.c:250:qmt_lqa_insert_range()) lustre-QMT0000: LQA range 15-31 partially overlaps with existing range 10-19: rc = -34 [15371.599240] LustreError: 271811:0:(qmt_lqa.c:669:qmt_lqa_add()) lustre-QMT0000: lqa:lqa1 can't add range 15:31: rc = -17 [15374.820639] LustreError: 272008:0:(qmt_lqa.c:825:qmt_lqa_remove()) lustre-QMT0000: lqa-md-lqa1 cannot remove range 10:19: rc = -2 [15385.882213] Lustre: DEBUG MARKER: adding 50 LQA ranges took 2s [15390.901422] Lustre: DEBUG MARKER: Removing 50 LQA ranges took 1s [15399.887882] Lustre: DEBUG MARKER: == sanity-quota test 97c: LQA lfs quota commands and MDS restart consistency ========================================================== 12:59:25 (1787849965) [15405.093365] Lustre: Failing over lustre-MDT0000 [15405.501572] Lustre: server umount lustre-MDT0000 complete [15415.307070] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15415.759797] Lustre: 273871:0:(scrub.c:697:lustre_index_register()) lustre-MDT0000: the index [0x200000003:0x74:0x0] has registered with 8/8, may be invalid, replace with 4/8 [15415.774096] LustreError: 273871:0:(qmt_lqa.c:522:qmt_lqa_load_ranges_from_disk()) lustre-QMT0000: Failed to get record from IAM iterator: rc = -2 [15415.832720] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15416.013604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15416.017335] Lustre: Skipped 1 previous similar message [15416.095789] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [15418.662436] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [15418.801665] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [15418.853702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:97) [15421.416145] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15421.432533] Lustre: Skipped 1 previous similar message [15421.516463] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15422.435773] Lustre: 3303:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849975/real 1787849975] req@ffff8d74af85b0c0 x1874680536796160/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849991 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [15422.482938] Lustre: 3303:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15427.556059] Lustre: 3300:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849980/real 1787849980] req@ffff8d74b5516d00 x1874680536796416/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [15427.776952] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [15436.092510] Lustre: DEBUG MARKER: == sanity-quota test 97d: LQA disk persistence using standard commands ========================================================== 13:00:01 (1787850001) [15455.681725] Lustre: Failing over lustre-MDT0000 [15456.012283] Lustre: server umount lustre-MDT0000 complete [15464.836809] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15465.261090] Lustre: 275923:0:(scrub.c:697:lustre_index_register()) lustre-MDT0000: the index [0x200000003:0x87:0x0] has registered with 8/8, may be invalid, replace with 4/8 [15465.270269] LustreError: 275923:0:(qmt_lqa.c:522:qmt_lqa_load_ranges_from_disk()) lustre-QMT0000: Failed to get record from IAM iterator: rc = -2 [15465.306114] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15465.317690] Lustre: Skipped 2 previous similar messages [15465.519520] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [15469.837744] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [15469.934104] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [15470.001598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:129) [15470.499778] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15470.562190] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15470.577814] Lustre: Skipped 1 previous similar message [15472.485789] Lustre: 3301:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850025/real 1787850025] req@ffff8d74b55c2940 x1874680536812160/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [15472.520158] Lustre: 3301:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15476.846357] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [15478.753279] Lustre: 3302:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850031/real 1787850031] req@ffff8d74b9b5c780 x1874680536812416/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [15478.802646] Lustre: 3302:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [15519.662377] Lustre: DEBUG MARKER: == sanity-quota test 97e: LQA add/remove should reject invalid ranges ========================================================== 13:01:25 (1787850085) [15534.199881] LustreError: 278276:0:(qmt_lqa.c:79:qmt_lqa_destroy()) lustre-QMT0000: cannot destroy lqa-dt-lqa1: rc = -2 [15534.206216] LustreError: 278276:0:(qmt_lqa.c:84:qmt_lqa_destroy()) lustre-QMT0000: cannot destroy lqa-md-lqa1: rc = -2 [15536.027787] Lustre: DEBUG MARKER: == sanity-quota test 98: Verify MDT retry on -EDQUOT during mkdir ========================================================== 13:01:41 (1787850101) [15538.255096] Lustre: DEBUG MARKER: SKIP: sanity-quota test_98 needs >= 2 MDTs [15540.742376] Lustre: DEBUG MARKER: == sanity-quota test 300: inode quota with MDT directory migration at 80% limit ========================================================== 13:01:45 (1787850105) [15542.434842] Lustre: DEBUG MARKER: SKIP: sanity-quota test_300 needs >= 2 MDTs [15547.727552] Lustre: DEBUG MARKER: == sanity-quota test complete, duration 15289 sec ======== 13:01:53 (1787850113) [15549.773030] Lustre: DEBUG MARKER: === sanity-quota: start cleanup 13:01:55 (1787850115) === [15553.383651] Lustre: DEBUG MARKER: === sanity-quota: finish cleanup 13:01:59 (1787850119) === [15562.309685] Lustre: server umount lustre-MDT0000 complete [15567.038685] LustreError: 258357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850135 with bad export cookie 4273330425827350673 [15567.052536] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15576.670374] Lustre: server umount lustre-OST0000 complete [15578.596142] Lustre: 3302:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850131/real 1787850131] req@ffff8d74997cf0c0 x1874680536837632/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850147 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [15578.641244] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15581.913313] Lustre: server umount lustre-OST0001 complete [15595.499318] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [15598.632699] Key type lgssc unregistered [15598.936063] LNet: 279827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15598.953470] LNetError: 279827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15598.982472] LNet: Removed LNI 192.168.204.117@tcp [15600.241162] Key type .llcrypt unregistered [15600.248107] Key type ._llcrypt unregistered