[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 462223636 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002340] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006828] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013079] *** VALIDATE selinux *** [ 0.020561] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024274] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027111] *** VALIDATE tmpfs *** [ 0.028431] *** VALIDATE proc *** [ 0.030067] *** VALIDATE cgroup *** [ 0.031014] *** VALIDATE cgroup2 *** [ 0.033116] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036042] Spectre V2 : User space: Vulnerable [ 0.037012] Speculative Store Bypass: Vulnerable [ 0.039705] debug: unmapping init [mem 0xffffffff8f259000-0xffffffff8f260fff] [ 0.041174] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042678] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043028] ... version: 2 [ 0.044013] ... bit width: 48 [ 0.045009] ... generic registers: 4 [ 0.046009] ... value mask: 0000ffffffffffff [ 0.047016] ... max period: 00007fffffffffff [ 0.048015] ... fixed-purpose events: 3 [ 0.049012] ... event mask: 000000070000000f [ 0.050287] rcu: Hierarchical SRCU implementation. [ 0.052507] smp: Bringing up secondary CPUs ... [ 0.053591] x86: Booting SMP configuration: [ 0.054023] .... node #0, CPUs: #1 #2 #3 [ 0.057290] smp: Brought up 1 node, 4 CPUs [ 0.059015] smpboot: Max logical packages: 1 [ 0.060013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138402] node 0 deferred pages initialised in 76ms [ 0.142098] devtmpfs: initialized [ 0.144370] x86/mm: Memory block size: 128MB [ 0.148404] gcov: version magic: 0x41383552 [ 0.152305] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157126] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160350] pinctrl core: initialized pinctrl subsystem [ 0.163215] [ 0.163786] ************************************************************* [ 0.167013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170021] ** ** [ 0.172010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175013] ** ** [ 0.178015] ** This means that this kernel is built to expose internal ** [ 0.180012] ** IOMMU data structures, which may compromise security on ** [ 0.183013] ** your system. ** [ 0.185015] ** ** [ 0.189014] ** If you see this message and you are not debugging the ** [ 0.192011] ** kernel, report this immediately to your vendor! ** [ 0.194014] ** ** [ 0.197013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.200054] ************************************************************* [ 0.203585] NET: Registered protocol family 16 [ 0.204430] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.207057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.209059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.212030] cpuidle: using governor menu [ 0.213966] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.216511] PCI: Using configuration type 1 for base access [ 0.219146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228068] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.233091] cryptd: max_cpu_qlen set to 1000 [ 0.238170] ACPI: Added _OSI(Module Device) [ 0.240015] ACPI: Added _OSI(Processor Device) [ 0.243026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.245017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.251687] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.260396] ACPI: Interpreter enabled [ 0.263013] ACPI: PM: (supports S0 S3 S4 S5) [ 0.266038] ACPI: Using IOAPIC for interrupt routing [ 0.268103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.272405] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.283556] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.286041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.289020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.293088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.299421] acpiphp: Slot [2] registered [ 0.302159] acpiphp: Slot [5] registered [ 0.304191] acpiphp: Slot [6] registered [ 0.306116] acpiphp: Slot [7] registered [ 0.307121] acpiphp: Slot [8] registered [ 0.309122] acpiphp: Slot [9] registered [ 0.311121] acpiphp: Slot [10] registered [ 0.312145] acpiphp: Slot [3] registered [ 0.314226] acpiphp: Slot [4] registered [ 0.316143] acpiphp: Slot [11] registered [ 0.318145] acpiphp: Slot [12] registered [ 0.319140] acpiphp: Slot [13] registered [ 0.321145] acpiphp: Slot [14] registered [ 0.322224] acpiphp: Slot [15] registered [ 0.324132] acpiphp: Slot [16] registered [ 0.326166] acpiphp: Slot [17] registered [ 0.328308] acpiphp: Slot [18] registered [ 0.331167] acpiphp: Slot [19] registered [ 0.333164] acpiphp: Slot [20] registered [ 0.336091] acpiphp: Slot [21] registered [ 0.337083] acpiphp: Slot [22] registered [ 0.339088] acpiphp: Slot [23] registered [ 0.340117] acpiphp: Slot [24] registered [ 0.342109] acpiphp: Slot [25] registered [ 0.343095] acpiphp: Slot [26] registered [ 0.345107] acpiphp: Slot [27] registered [ 0.347070] acpiphp: Slot [28] registered [ 0.348077] acpiphp: Slot [29] registered [ 0.350073] acpiphp: Slot [30] registered [ 0.351076] acpiphp: Slot [31] registered [ 0.352043] PCI host bridge to bus 0000:00 [ 0.353008] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.354017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.356022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.358017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.360029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.363031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.365230] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.367926] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.372324] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.381739] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.387061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.390025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.392018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.395018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.398616] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.401956] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.404038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.407863] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.413016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.424992] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.430022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.436551] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.452013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.470014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.520016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.542856] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.557017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.569027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.599019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.613943] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.624025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.632026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.654022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.668507] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.675025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.683020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.706023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.716603] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.724025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.732024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.754022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.770127] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.777026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.791015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.817017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.827620] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.829463] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.832392] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.836318] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.838342] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.845115] iommu: Default domain type: Passthrough [ 0.846639] SCSI subsystem initialized [ 0.849173] ACPI: bus type USB registered [ 0.850127] usbcore: registered new interface driver usbfs [ 0.852077] usbcore: registered new interface driver hub [ 0.854097] usbcore: registered new device driver usb [ 0.856195] pps_core: LinuxPPS API ver. 1 registered [ 0.858013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.861091] PTP clock support registered [ 0.864138] EDAC MC: Ver: 3.0.0 [ 0.865485] PCI: Using ACPI for IRQ routing [ 0.867929] NetLabel: Initializing [ 0.869013] NetLabel: domain hash size = 128 [ 0.871013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.873152] NetLabel: unlabeled traffic allowed by default [ 0.875390] vgaarb: loaded [ 0.877328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.879032] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.887869] clocksource: Switched to clocksource kvm-clock [ 0.996111] VFS: Disk quotas dquot_6.6.0 [ 0.997904] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.001157] *** VALIDATE ramfs *** [ 1.002442] *** VALIDATE hugetlbfs *** [ 1.004312] pnp: PnP ACPI init [ 1.006767] pnp: PnP ACPI: found 6 devices [ 1.027197] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.031855] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.034885] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.037028] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.039293] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.041641] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.044213] NET: Registered protocol family 2 [ 1.046766] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.052596] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.055877] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.060955] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.064519] TCP: Hash tables configured (established 65536 bind 65536) [ 1.067722] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.070541] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.074637] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.078295] NET: Registered protocol family 1 [ 1.082181] RPC: Registered named UNIX socket transport module. [ 1.084665] RPC: Registered udp transport module. [ 1.087127] RPC: Registered tcp transport module. [ 1.088313] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.089932] NET: Registered protocol family 44 [ 1.091404] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.093845] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.096382] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.099154] PCI: CLS 0 bytes, default 64 [ 1.100636] Unpacking initramfs... [ 4.033898] debug: unmapping init [mem 0xffffa0fc3cc54000-0xffffa0fc3ffbffff] [ 4.050323] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.055119] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.070760] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.045478] Initialise system trusted keyrings [ 6.062150] Key type blacklist registered [ 6.064621] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.082166] zbud: loaded [ 6.090975] *** VALIDATE nfs *** [ 6.092976] *** VALIDATE nfs4 *** [ 6.098796] pstore: using deflate compression [ 6.116752] Platform Keyring initialized [ 6.398371] NET: Registered protocol family 38 [ 6.405573] Key type asymmetric registered [ 6.415360] Asymmetric key parser 'x509' registered [ 6.417338] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.423724] io scheduler mq-deadline registered [ 6.431957] io scheduler kyber registered [ 6.438247] io scheduler bfq registered [ 6.442266] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.457708] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.465986] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.474571] ACPI: Power Button [PWRF] [ 6.493078] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.521949] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.571707] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.587318] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.616424] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.654169] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.692896] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.702585] Non-volatile memory driver v1.3 [ 6.705921] Linux agpgart interface v0.103 [ 6.778300] virtio_blk virtio1: [vda] 146944 512-byte logical blocks (75.2 MB/71.8 MiB) [ 6.795525] vda: detected capacity change from 0 to 75235328 [ 6.831241] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.841291] vdb: detected capacity change from 0 to 1073741824 [ 6.893402] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.909687] vdc: detected capacity change from 0 to 2621440000 [ 6.975279] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.985328] vdd: detected capacity change from 0 to 2621440000 [ 7.034025] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.049182] vde: detected capacity change from 0 to 4294967296 [ 7.124019] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.136245] vdf: detected capacity change from 0 to 4294967296 [ 7.154329] libphy: Fixed MDIO Bus: probed [ 7.185248] usbcore: registered new interface driver usbserial_generic [ 7.206896] usbserial: USB Serial support registered for generic [ 7.212317] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.244194] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.246242] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.259332] mousedev: PS/2 mouse device common for all mice [ 7.273453] rtc_cmos 00:05: RTC can wake from S4 [ 7.275223] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.289235] rtc_cmos 00:05: registered as rtc0 [ 7.293437] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.295882] intel_pstate: CPU model not supported [ 7.301376] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.320802] hid: raw HID events driver (C) Jiri Kosina [ 7.324403] usbcore: registered new interface driver usbhid [ 7.328142] usbhid: USB HID core driver [ 7.330122] drop_monitor: Initializing network drop monitor service [ 7.332112] Initializing XFRM netlink socket [ 7.335859] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.352035] NET: Registered protocol family 10 [ 7.356785] Segment Routing with IPv6 [ 7.358308] NET: Registered protocol family 17 [ 7.364152] mpls_gso: MPLS GSO support [ 7.378637] RAS: Correctable Errors collector initialized. [ 7.382097] AVX version of gcm_enc/dec engaged. [ 7.389273] AES CTR mode by8 optimization enabled [ 7.608331] sched_clock: Marking stable (7608251067, 0)->(8501115934, -892864867) [ 7.627653] registered taskstats version 1 [ 7.632762] Loading compiled-in X.509 certificates [ 7.637038] zswap: loaded using pool lzo/zbud [ 7.730975] Key type big_key registered [ 7.748901] Key type encrypted registered [ 7.750822] ima: No TPM chip found, activating TPM-bypass! [ 7.753917] ima: Allocated hash algorithm: sha1 [ 7.756567] ima: No architecture policies found [ 7.761972] evm: Initialising EVM extended attributes: [ 7.765222] evm: security.selinux [ 7.766901] evm: security.ima [ 7.768741] evm: security.capability [ 7.769883] evm: HMAC attrs: 0x1 [ 7.774594] rtc_cmos 00:05: setting system clock to 2026-08-27 14:31:05 UTC (1787841065) [ 7.790677] debug: unmapping init [mem 0xffffffff90203000-0xffffffff903fffff] [ 7.796635] debug: unmapping init [mem 0xffffffff8ef82000-0xffffffff8f258fff] [ 7.806268] Write protecting the kernel read-only data: 28672k [ 7.815317] debug: unmapping init [mem 0xffffffff8d603000-0xffffffff8d7fffff] [ 7.824548] debug: unmapping init [mem 0xffffffff8df14000-0xffffffff8dffffff] [ 7.950683] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.978876] systemd[1]: Detected virtualization kvm. [ 7.985318] systemd[1]: Detected architecture x86-64. [ 7.996383] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.128451] systemd[1]: No hostname configured. [ 8.133935] systemd[1]: Set hostname to . [ 8.142222] random: systemd: uninitialized urandom read (16 bytes read) [ 8.149866] systemd[1]: Initializing machine ID from random generator. [ 8.343139] random: ln: uninitialized urandom read (6 bytes read) [ 8.557248] random: systemd: uninitialized urandom read (16 bytes read) [ 8.573071] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 8.583427] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 8.614972] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.028804] device-mapper: uevent: version 1.0.3 [ 10.040496] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.484446] virtio_net virtio0 ens2: renamed from eth0 [ 11.604272] random: fast init done [ 11.794263] scsi host0: ata_piix [ 11.808199] scsi host1: ata_piix [ 11.810743] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.814914] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.283661] random: crng init done [ 16.285252] random: 7 urandom warning(s) missed due to ratelimiting [ 17.994554] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.580648] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.815701] printk: systemd: 25 output lines suppressed due to ratelimiting [ 23.627478] SELinux: Disabled at runtime. [ 23.750445] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.774914] systemd[1]: Detected virtualization kvm. [ 23.777919] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.570335] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.575627] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.585762] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.592801] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.597923] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.625130] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.637960] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ 26.014771] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 27.047834] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.931766] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.050738] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.400845] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.441509] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 33.519595] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 34.610809] NFS: Registering the id_resolver key type [ 34.617697] Key type id_resolver registered [ 34.623188] Key type id_legacy registered [ ***] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] [ 38.933007] hrtimer: interrupt took 4105618 ns Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg325-server login: [ 80.922037] spl: loading out-of-tree module taints kernel. [ 87.213259] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 102.427538] Key type ._llcrypt registered [ 102.429027] Key type .llcrypt registered [ 102.602124] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_hostid [ 119.620240] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [ 120.550040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 120.581978] alg: No test for adler32 (adler32-zlib) [ 122.110839] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 123.311460] LNet: Added LNI 192.168.203.125@tcp [8/256/0/180] [ 125.167334] Key type lgssc registered [ 126.737240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.013415] vdc: vdc1 vdc9 [ 146.800707] vde: vde1 vde9 [ 156.993278] vdf: vdf1 vdf9 [ 178.661348] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [ 188.877380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 190.096907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 190.336210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 190.413431] Lustre: lustre-MDT0000: new disk, initializing [ 190.792355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.872474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.400534] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.471729] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 207.636280] Lustre: lustre-OST0000: new disk, initializing [ 207.643419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 207.650865] Lustre: Skipped 1 previous similar message [ 207.737957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.665914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 210.689801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 210.911477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 214.418622] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.864335] Lustre: lustre-OST0001: new disk, initializing [ 224.875698] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 224.965357] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.038685] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.105686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 234.112606] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 234.198412] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 245.416883] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 254.698177] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 260.762447] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.689957] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing yml_node [ 270.303824] Lustre: DEBUG MARKER: Client: 2.17.57.97 [ 273.164452] Lustre: DEBUG MARKER: MDS: 2.17.57.97 [ 275.622837] Lustre: DEBUG MARKER: OSS: 2.17.57.97 [ 277.573977] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-quota ============----- Thu Aug 27 10:35:33 EDT 2026 [ 297.960171] Lustre: DEBUG MARKER: excepting tests: 2 4a 63 65 [ 299.594813] Lustre: DEBUG MARKER: skipping tests SLOW=no: 61 12a 9 [ 302.775660] Lustre: DEBUG MARKER: === sanity-quota: start setup 10:35:59 (1787841359) === [ 308.229255] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing check_config_client /mnt/lustre [ 323.614423] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 327.827862] Lustre: 11349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 333.615259] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 339.310699] Lustre: DEBUG MARKER: === sanity-quota: finish setup 10:36:34 (1787841394) === [ 396.898761] Lustre: DEBUG MARKER: == sanity-quota test 0: Test basic quota performance ===== 10:37:33 (1787841453) [ 454.303072] Lustre: DEBUG MARKER: == sanity-quota test 1a: Block hard limit (normal use and out of quota) ========================================================== 10:38:30 (1787841510) [ 470.984850] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 477.649840] Lustre: DEBUG MARKER: Write... [ 479.727681] Lustre: DEBUG MARKER: Write out of block quota ... [ 519.037400] Lustre: DEBUG MARKER: -------------------------------------- [ 520.365025] Lustre: DEBUG MARKER: Group quota (block hardlimit:10 MB) [ 526.980569] Lustre: DEBUG MARKER: Write... [ 529.208151] Lustre: DEBUG MARKER: Write out of block quota ... [ 575.825874] Lustre: DEBUG MARKER: -------------------------------------- [ 577.155764] Lustre: DEBUG MARKER: Project quota (block hardlimit:10 mb) [ 579.992489] Lustre: DEBUG MARKER: Write... [ 582.522990] Lustre: DEBUG MARKER: Write out of block quota ... [ 646.072660] Lustre: DEBUG MARKER: == sanity-quota test 1b: Quota pools: Block hard limit (normal use and out of quota) ========================================================== 10:41:42 (1787841702) [ 663.963089] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 683.137526] Lustre: DEBUG MARKER: Write... [ 685.648322] Lustre: DEBUG MARKER: Write out of block quota ... [ 729.717845] Lustre: DEBUG MARKER: -------------------------------------- [ 731.579563] Lustre: DEBUG MARKER: Group quota (block hardlimit:20 MB) [ 739.427962] Lustre: DEBUG MARKER: Write... [ 742.056485] Lustre: DEBUG MARKER: Write out of block quota ... [ 787.820513] Lustre: DEBUG MARKER: -------------------------------------- [ 789.504474] Lustre: DEBUG MARKER: Project quota (block hardlimit:20 mb) [ 792.496534] Lustre: DEBUG MARKER: Write... [ 795.510321] Lustre: DEBUG MARKER: Write out of block quota ... [ 874.638142] Lustre: DEBUG MARKER: == sanity-quota test 1c: Quota pools: check 3 pools with hardlimit only for global ========================================================== 10:45:30 (1787841930) [ 892.945068] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 917.610711] Lustre: DEBUG MARKER: Write... [ 921.014859] Lustre: DEBUG MARKER: Write out of block quota ... [ 1019.355844] Lustre: DEBUG MARKER: == sanity-quota test 1d: Quota pools: check block hardlimit on different pools ========================================================== 10:47:55 (1787842075) [ 1037.923382] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 1061.047737] Lustre: DEBUG MARKER: Write... [ 1064.043513] Lustre: DEBUG MARKER: Write out of block quota ... [ 1156.761419] Lustre: DEBUG MARKER: == sanity-quota test 1e: Quota pools: global pool high block limit vs quota pool with small ========================================================== 10:50:13 (1787842213) [ 1173.432729] Lustre: DEBUG MARKER: User quota (block hardlimit:53000000 MB) [ 1186.072497] Lustre: DEBUG MARKER: Write... [ 1188.591372] Lustre: DEBUG MARKER: Write out of block quota ... [ 1201.207547] Lustre: DEBUG MARKER: Write... [ 1263.948616] Lustre: DEBUG MARKER: == sanity-quota test 1f: Quota pools: correct qunit after removing/adding OST ========================================================== 10:52:00 (1787842320) [ 1279.623709] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 1291.325345] Lustre: DEBUG MARKER: Write... [ 1293.913685] Lustre: DEBUG MARKER: Write out of block quota ... [ 1338.244168] Lustre: DEBUG MARKER: Write... [ 1340.948813] Lustre: DEBUG MARKER: Write out of block quota ... [ 1392.731095] Lustre: DEBUG MARKER: == sanity-quota test 1g: Quota pools: Block hard limit with wide striping ========================================================== 10:54:09 (1787842449) [ 1409.189127] Lustre: DEBUG MARKER: User quota (block hardlimit:40 MB) [ 1424.278343] Lustre: DEBUG MARKER: Write... [ 1435.268914] Lustre: DEBUG MARKER: Write out of block quota ... [ 1524.019146] Lustre: DEBUG MARKER: == sanity-quota test 1h: Block hard limit test using fallocate ========================================================== 10:56:20 (1787842580) [ 1525.348083] Lustre: DEBUG MARKER: SKIP: sanity-quota test_1h need >= 2.13.57 and ldiskfs for fallocate [ 1526.728335] Lustre: DEBUG MARKER: == sanity-quota test 1i: Quota pools: different limit and usage relations ========================================================== 10:56:23 (1787842583) [ 1542.064701] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 1555.492416] Lustre: DEBUG MARKER: Write... [ 1557.944912] Lustre: DEBUG MARKER: Write out of block quota ... [ 1595.871682] Lustre: DEBUG MARKER: Write... [ 1598.662753] Lustre: DEBUG MARKER: Write out of block quota ... [ 1610.350076] LustreError: 5845:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:15366 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 1660.841812] Lustre: DEBUG MARKER: == sanity-quota test 1j: Enable project quota enforcement for root ========================================================== 10:58:37 (1787842717) [ 1678.202635] Lustre: DEBUG MARKER: -------------------------------------- [ 1679.595507] Lustre: DEBUG MARKER: Project quota (hardlimit: 40 mb, 2048 files) [ 2002.675755] Lustre: DEBUG MARKER: == sanity-quota test 1k: LQA: Block hard limit (normal use and out of quota) ========================================================== 11:04:19 (1787843059) [ 2020.901417] Lustre: DEBUG MARKER: Write... [ 2023.905529] Lustre: DEBUG MARKER: Write out of block quota ... [ 2060.334699] LustreError: 3315:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-OST0001: Delaying reintegration for qtype:1 until pending updates are flushed. [ 2062.579925] Lustre: DEBUG MARKER: Write... [ 2065.504383] Lustre: DEBUG MARKER: Write out of block quota ... [ 2105.781142] Lustre: DEBUG MARKER: Write... [ 2109.170421] Lustre: DEBUG MARKER: Write out of block quota ... [ 2157.450409] Lustre: DEBUG MARKER: == sanity-quota test 1l: Async writes should not be rejected by quota with root_prj_enable ========================================================== 11:06:54 (1787843214) [ 2215.508235] Lustre: DEBUG MARKER: SKIP: sanity-quota test_2 skipping excluded test 2 [ 2216.956660] Lustre: DEBUG MARKER: == sanity-quota test 3a: Block soft limit (start timer, timer goes off, stop timer) ========================================================== 11:07:53 (1787843273) [ 2305.778925] Lustre: DEBUG MARKER: Write after timer goes off [ 2307.734873] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2444.720831] Lustre: DEBUG MARKER: Write after timer goes off [ 2446.716476] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2583.507562] Lustre: DEBUG MARKER: Write after timer goes off [ 2585.395645] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2690.510428] Lustre: DEBUG MARKER: == sanity-quota test 3b: Quota pools: Block soft limit (start timer, expires, stop timer) ========================================================== 11:15:47 (1787843747) [ 2785.409346] Lustre: DEBUG MARKER: Write after timer goes off [ 2787.179734] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2924.392441] Lustre: DEBUG MARKER: Write after timer goes off [ 2925.989856] Lustre: DEBUG MARKER: Write after cancel lru locks [ 3056.394252] Lustre: DEBUG MARKER: Write after timer goes off [ 3058.048275] Lustre: DEBUG MARKER: Write after cancel lru locks [ 3175.190588] Lustre: DEBUG MARKER: == sanity-quota test 3c: Quota pools: check block soft limit on different pools ========================================================== 11:23:51 (1787844231) [ 3276.498132] Lustre: DEBUG MARKER: Write after timer goes off [ 3278.180453] Lustre: DEBUG MARKER: Write after cancel lru locks [ 3375.416871] Lustre: DEBUG MARKER: SKIP: sanity-quota test_4a skipping excluded test 4a [ 3377.173730] Lustre: DEBUG MARKER: == sanity-quota test 4b: Grace time strings handling ===== 11:27:13 (1787844433) [ 3394.838632] Lustre: DEBUG MARKER: == sanity-quota test 5: Chown [ 3520.162514] Lustre: DEBUG MARKER: == sanity-quota test 6: Test dropping acquire request on master ========================================================== 11:29:36 (1787844576) [ 3578.335925] Lustre: *** cfs_fail_loc=513, val=601*** [ 3578.340897] Lustre: Skipped 2 previous similar messages [ 3579.361164] Lustre: *** cfs_fail_loc=513, val=601*** [ 3580.426815] Lustre: *** cfs_fail_loc=513, val=601*** [ 3580.430492] Lustre: Skipped 13 previous similar messages [ 3580.435373] LustreError: 14374:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1874687355986560 [ 3580.454703] LustreError: 14374:0:(service.c:2341:ptlrpc_server_handle_req_in()) Skipped 2 previous similar messages [ 3582.433155] Lustre: *** cfs_fail_loc=513, val=601*** [ 3582.438419] Lustre: Skipped 8 previous similar messages [ 3586.680755] Lustre: *** cfs_fail_loc=513, val=601*** [ 3586.686115] Lustre: Skipped 9 previous similar messages [ 3594.719711] Lustre: *** cfs_fail_loc=513, val=601*** [ 3594.724538] Lustre: Skipped 18 previous similar messages [ 3596.767170] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844638/real 1787844638] req@ffffa0fb896a5e00 x1874687355986688/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787844654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 3596.768614] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.807289] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3596.836315] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 3596.848642] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3598.947464] LustreError: 5850:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1874687355991296 [ 3605.472580] LustreError: 14369:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1874687355992832 [ 3605.487899] LustreError: 14369:0:(service.c:2341:ptlrpc_server_handle_req_in()) Skipped 2 previous similar messages [ 3612.128215] Lustre: *** cfs_fail_loc=513, val=601*** [ 3612.137740] Lustre: Skipped 43 previous similar messages [ 3615.199217] Lustre: 6692:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844656/real 1787844656] req@ffffa0fb878970c0 x1874687355991296/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787844672 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_001.0' uid:0 gid:0 projid:4294967295 [ 3615.232338] Lustre: 6692:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3615.250769] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.265969] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 3615.276778] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3618.332715] LustreError: 5850:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1874687355995904 [ 3621.344175] Lustre: 3317:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844663/real 1787844663] req@ffffa0fb89460f00 x1874687355992576/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787844679 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295 [ 3621.376904] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3621.386741] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0001_UUID (at 0@lo) reconnecting [ 3621.400867] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3634.656566] Lustre: 39105:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844676/real 1787844676] req@ffffa0fb87eb4000 x1874687355995904/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787844692 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_005.0' uid:0 gid:0 projid:4294967295 [ 3634.690753] Lustre: 39105:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3634.708061] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.731282] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 3634.749697] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3677.821554] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 11:32:14 (1787844734) [ 3708.746348] Lustre: Failing over lustre-OST0000 [ 3709.608646] Lustre: lustre-OST0000: Not available for connect from 192.168.203.25@tcp (stopping) [ 3710.510792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 3710.524405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.536829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3711.032247] Lustre: server umount lustre-OST0000 complete [ 3716.577885] LustreError: 39049:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3716.618950] LustreError: 39049:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3719.808540] LustreError: 9731:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.714252] LustreError: 39101:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3723.207502] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 3723.227564] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3724.646952] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3724.863925] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3724.870189] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3729.398717] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3737.004902] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3742.144140] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3747.919570] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3753.456530] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3759.747469] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3765.633699] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3796.246733] Lustre: Failing over lustre-OST0000 [ 3796.435331] Lustre: server umount lustre-OST0000 complete [ 3796.451165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3796.456206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.475471] LustreError: 6687:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3805.156564] LustreError: 39050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3805.178057] LustreError: 39050:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3806.693127] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 3806.719394] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3808.226458] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3808.367165] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3808.383067] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3813.660437] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3821.147635] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3826.291094] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3831.111951] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3836.671357] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3842.669353] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3848.872398] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3884.854937] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) ========================================================== 11:35:41 (1787844941) [ 3919.933368] Lustre: *** cfs_fail_loc=a02, val=0*** [ 3927.960886] Lustre: Failing over lustre-OST0000 [ 3928.079334] Lustre: server umount lustre-OST0000 complete [ 3929.570664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3929.592868] LustreError: 6687:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3936.497197] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 3936.531761] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3937.678416] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3938.150249] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3938.151220] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3944.201518] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3953.102139] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3959.344912] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3965.822625] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3973.059826] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3979.133937] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 3984.903040] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4016.048846] LustreError: 3315:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-OST0000: Delaying reintegration for qtype:0 until pending updates are flushed. [ 4038.183939] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ========================================================== 11:38:14 (1787845094) [ 4065.647630] LustreError: 91427:0:(qsd_reint.c:488:qsd_reint_main()) cfs_fail_timeout id a03 sleeping for 10000ms [ 4065.656523] LustreError: 91427:0:(qsd_reint.c:488:qsd_reint_main()) Skipped 1 previous similar message [ 4067.338353] Lustre: Failing over lustre-MDT0000 [ 4067.722322] Lustre: server umount lustre-MDT0000 complete [ 4070.680166] LustreError: 91430:0:(qsd_reint.c:488:qsd_reint_main()) cfs_fail_timeout interrupted [ 4076.489185] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4076.883467] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4076.898818] Lustre: Skipped 1 previous similar message [ 4077.067232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4077.162501] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4078.233498] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4078.387186] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4078.443223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:161 to 0x240000400:193) [ 4078.444201] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 4081.411839] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4082.153982] LustreError: 3314:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x2020000:0x0].0x0 (ffffa0fc90ed9300) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4082.176129] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4085.414972] Lustre: 3318:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845127/real 1787845127] req@ffffa0fb977143c0 x1874687356360832/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787845143 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4090.086982] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4185.250871] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4190.673871] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4196.342309] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4202.030820] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4207.110815] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4236.140258] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ========================================================== 11:41:33 (1787845293) [ 4266.287869] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 4271.641623] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 4318.679717] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================================== 11:42:55 (1787845375) [ 4319.917397] Lustre: DEBUG MARKER: SKIP: sanity-quota test_7e needs >= 2 MDTs [ 4321.320768] Lustre: DEBUG MARKER: == sanity-quota test 7f: Quota reintegration automatically ========================================================== 11:42:58 (1787845378) [ 4335.182529] Lustre: *** cfs_fail_loc=a11, val=0*** [ 4335.187171] Lustre: Skipped 2 previous similar messages [ 4341.164138] Lustre: *** cfs_fail_loc=a11, val=0*** [ 4396.015180] Lustre: *** cfs_fail_loc=a11, val=0*** [ 4396.029351] Lustre: Skipped 1 previous similar message [ 4524.112878] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ==== 11:46:20 (1787845580) [ 4730.551688] Lustre: DEBUG MARKER: SKIP: sanity-quota test_9 skipping SLOW test 9 [ 4732.253885] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ======== 11:49:48 (1787845788) [ 4782.836895] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota ======= 11:50:39 (1787845839) [ 4837.924966] Lustre: DEBUG MARKER: SKIP: sanity-quota test_12a skipping SLOW test 12a [ 4839.554625] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ======== 11:51:36 (1787845896) [ 4840.904508] Lustre: DEBUG MARKER: SKIP: sanity-quota test_12b needs >= 2 MDTs [ 4842.540363] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================================== 11:51:39 (1787845899) [ 4911.641512] Lustre: DEBUG MARKER: == sanity-quota test 14: check panic in qmt_site_recalc_cb ========================================================== 11:52:47 (1787845967) [ 4937.331959] Lustre: Failing over lustre-OST0000 [ 4937.442054] Lustre: server umount lustre-OST0000 complete [ 4938.408355] LustreError: 9731:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4938.441955] LustreError: 9731:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4942.326205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4942.362041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4942.366203] LustreError: 6685:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.431038] LustreError: 101941:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.451521] LustreError: 101941:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4949.969553] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4950.003088] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4951.160735] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4951.314345] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4951.323593] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4951.339990] Lustre: Skipped 1 previous similar message [ 4956.378381] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4995.855337] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ========= 11:54:12 (1787846052) [ 5025.951244] Lustre: DEBUG MARKER: == sanity-quota test 16a: lfs quota should skip the inactive MDT/OST ========================================================== 11:54:42 (1787846082) [ 5052.216475] Lustre: lustre-OST0000: Client 19e0ae59-0fc8-4940-9bc2-ce19a2cd5bf0 (at 192.168.203.25@tcp) reconnecting [ 5080.528493] Lustre: DEBUG MARKER: == sanity-quota test 16b: lfs quota should skip the nonexistent MDT/OST ========================================================== 11:55:37 (1787846137) [ 5082.240527] Lustre: DEBUG MARKER: SKIP: sanity-quota test_16b needs >= 3 MDTs [ 5084.176934] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error == 11:55:40 (1787846140) [ 5111.080660] Lustre: *** cfs_fail_loc=a04, val=37*** [ 5111.094129] LustreError: 39330:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x1 qsd:lustre-OST0000 qtype:usr lqe: ffffa0fca58e7200 id:60000 enforced:1 granted: 0 pending:0 waiting:1 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 [ 5112.243895] Lustre: *** cfs_fail_loc=a04, val=37*** [ 5112.249507] LustreError: 101894:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x1 qsd:lustre-OST0000 qtype:usr lqe: ffffa0fca58e7200 id:60000 enforced:1 granted: 0 pending:0 waiting:1024 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 [ 5173.866969] Lustre: *** cfs_fail_loc=a04, val=11*** [ 5238.880825] Lustre: *** cfs_fail_loc=a04, val=110*** [ 5238.886228] Lustre: Skipped 1 previous similar message [ 5304.377721] Lustre: *** cfs_fail_loc=a04, val=107*** [ 5304.384800] Lustre: Skipped 2 previous similar messages [ 5410.014432] Lustre: DEBUG MARKER: == sanity-quota test 18: MDS failover while writing, no watchdog triggered (b14840) ========================================================== 12:01:06 (1787846466) [ 5429.430659] Lustre: DEBUG MARKER: User quota (limit: 200) [ 5435.238396] Lustre: DEBUG MARKER: Write 100M (buffered) ... [ 5438.330109] LustreError: 122001:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 5439.180325] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5440.874480] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 5443.184893] Lustre: Failing over lustre-MDT0000 [ 5443.241869] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107 [ 5443.252703] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.282176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5443.630106] Lustre: server umount lustre-MDT0000 complete [ 5461.236215] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5461.528197] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5461.716927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5461.786750] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5462.695354] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5462.714177] Lustre: 3318:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846504/real 1787846504] req@ffffa0fc9eb01a40 x1874687357567232/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846520 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5462.739463] Lustre: 3318:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5463.516943] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5463.649604] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5463.683327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:660 to 0x240000400:705) [ 5463.686109] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:592 to 0x280000400:609) [ 5466.026390] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5467.621135] Lustre: 3316:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846509/real 1787846509] req@ffffa0fb88e37c00 x1874687357567872/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5472.289108] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846514/real 1787846514] req@ffffa0fc9eb03480 x1874687357568384/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846530 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5475.661533] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5477.727923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5483.007898] Lustre: DEBUG MARKER: (dd_pid=115013, time=0, timeout=600) [ 5532.170589] Lustre: DEBUG MARKER: User quota (limit: 200) [ 5541.514178] Lustre: DEBUG MARKER: Write 100M (directio) ... [ 5544.336801] LustreError: 124862:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 5545.417942] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5547.171181] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 5549.586924] Lustre: Failing over lustre-MDT0000 [ 5549.854425] Lustre: server umount lustre-MDT0000 complete [ 5565.409033] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846607/real 1787846607] req@ffffa0fca5041a40 x1874687357606144/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5565.419226] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5565.439734] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5565.466607] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5576.385325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5576.528964] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5577.120310] Lustre: 3318:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846618/real 1787846618] req@ffffa0fca5042580 x1874687357607040/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846634 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5577.147298] Lustre: 3318:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 5577.857107] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5578.077401] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5578.171837] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:592 to 0x280000400:641) [ 5578.174509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737) [ 5581.837130] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5590.580524] LustreError: 3314:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffffa0fc95b69c00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5590.608185] LustreError: 3314:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 5 previous similar messages [ 5590.630527] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5590.638875] Lustre: Skipped 1 previous similar message [ 5592.718822] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5595.133683] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5612.432420] Lustre: DEBUG MARKER: (dd_pid=117409, time=10, timeout=600) [ 5679.902400] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 12:05:36 (1787846736) [ 5703.272910] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 5705.324731] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 5711.122526] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 5712.485219] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 5745.122832] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 12:06:41 (1787846801) [ 5769.441636] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [ 5785.837881] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [ 5787.404475] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [ 5789.337954] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000 [ 5791.698637] Lustre: DEBUG MARKER: Set quota for 1 times [ 5795.593551] Lustre: DEBUG MARKER: Set quota for 2 times [ 5799.627318] Lustre: DEBUG MARKER: Set quota for 3 times [ 5804.211343] Lustre: DEBUG MARKER: Set quota for 4 times [ 5807.681391] Lustre: DEBUG MARKER: Set quota for 5 times [ 5811.548900] Lustre: DEBUG MARKER: Set quota for 6 times [ 5815.601467] Lustre: DEBUG MARKER: Set quota for 7 times [ 5819.599488] Lustre: DEBUG MARKER: Set quota for 8 times [ 5884.141326] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 12:09:00 (1787846940) [ 5902.820176] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5902.823152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5902.844518] Lustre: Skipped 2 previous similar messages [ 5902.860855] Lustre: Skipped 1 previous similar message [ 5907.026884] Lustre: server umount lustre-MDT0000 complete [ 5911.195954] LustreError: 5832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846968 with bad export cookie 6482986910228965368 [ 5911.196121] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5911.207253] LustreError: 5832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5911.301447] Lustre: server umount lustre-OST0000 complete [ 5915.158147] Lustre: server umount lustre-OST0001 complete [ 5929.853368] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [ 5940.658836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5945.361638] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5948.419747] Lustre: 134680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5955.783108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5962.384912] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5965.992122] LustreError: 135045:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.028741] LustreError: 135045:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5966.053917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:739 to 0x240000400:769) [ 5971.427233] LustreError: 135045:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5971.961934] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5977.092407] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:673) [ 5981.002927] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5990.293843] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5993.975229] Lustre: 136606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6010.854742] Lustre: server umount lustre-MDT0000 complete [ 6015.671623] LustreError: 137014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847073 with bad export cookie 6482986910228972165 [ 6015.688783] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6026.344870] Lustre: server umount lustre-OST0000 complete [ 6029.279857] Lustre: 3317:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787847070/real 1787847070] req@ffffa0fcb721bc00 x1874687357775872/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787847086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6029.333684] Lustre: 3317:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6031.712376] Lustre: server umount lustre-OST0001 complete [ 6048.060961] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [ 6058.210424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.491831] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6065.531821] Lustre: 139261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6078.014778] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6082.979923] LustreError: 139625:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6083.009729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:739 to 0x240000400:801) [ 6085.866769] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6085.870455] Lustre: Skipped 1 previous similar message [ 6091.261479] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:705) [ 6091.516146] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6099.350708] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6103.244707] Lustre: 141175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6115.566512] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 12:12:51 (1787847171) [ 6117.123811] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS [ 6119.362582] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 12:12:55 (1787847175) [ 6178.558898] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 12:13:55 (1787847235) [ 6227.788614] Lustre: DEBUG MARKER: Write... [ 6230.176119] Lustre: DEBUG MARKER: Write out of block quota ... [ 6290.425287] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 12:15:46 (1787847346) [ 6297.928687] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 12:15:54 (1787847354) [ 6309.211412] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 12:16:05 (1787847365) [ 6319.192983] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 12:16:15 (1787847375) [ 6328.906601] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 12:16:25 (1787847385) [ 6397.904654] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user [ 6554.283888] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user [ 6741.237540] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 12:23:17 (1787847797) [ 6802.759041] Lustre: DEBUG MARKER: Restart... [ 6806.944561] Lustre: server umount lustre-MDT0000 complete [ 6810.596605] LustreError: 138816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847868 with bad export cookie 6482986910228973691 [ 6810.609894] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6820.965970] Lustre: server umount lustre-OST0000 complete [ 6823.393239] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787847865/real 1787847865] req@ffffa0fcbe8e9680 x1874687358003328/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787847881 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6823.417696] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6823.428957] Lustre: Skipped 1 previous similar message [ 6824.930991] Lustre: server umount lustre-OST0001 complete [ 6840.318752] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [ 6850.702057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6855.833746] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6859.146425] Lustre: 159626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6865.438262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6870.940713] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6875.629767] LustreError: 160559:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6875.646979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:810 to 0x240000400:833) [ 6879.006852] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6884.344941] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:714 to 0x280000400:737) [ 6884.546097] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6891.800219] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6894.954982] Lustre: 161551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6960.650760] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 12:26:57 (1787848017) [ 7035.110312] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 12:28:11 (1787848091) [ 8648.019458] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 12:55:04 (1787849704) [ 8661.699646] Lustre: server umount lustre-MDT0000 complete [ 8665.060862] LustreError: 159182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849722 with bad export cookie 6482986910228980194 [ 8665.071709] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8675.428705] Lustre: server umount lustre-OST0000 complete [ 8689.709809] Lustre: server umount lustre-OST0001 complete [ 8702.543374] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [ 8711.610713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8715.575595] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8718.160511] Lustre: 170024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8724.109881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8729.598873] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8736.299522] LustreError: 170389:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8736.324973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5836 to 0x240000400:5857) [ 8737.049764] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8742.410983] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5738 to 0x280000400:5761) [ 8743.121546] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8749.460178] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8752.490765] Lustre: 171945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8784.167514] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 12:57:20 (1787849840) [ 8831.092667] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 12:58:07 (1787849887) [ 8877.207906] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 12:58:53 (1787849933) [ 8878.585893] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs [ 8880.719749] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 12:58:57 (1787849937) [ 8882.711252] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs [ 8884.777609] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 12:59:01 (1787849941) [ 8973.406615] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 13:00:29 (1787850029) [ 9024.187837] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 13:01:20 (1787850080) [ 9116.701667] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 13:02:52 (1787850172) [ 9129.012354] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9129.017386] Lustre: Skipped 2 previous similar messages [ 9130.016309] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9130.018540] Lustre: Skipped 37 previous similar messages [ 9132.031381] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9132.037055] Lustre: Skipped 101 previous similar messages [ 9136.062303] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9136.066326] Lustre: Skipped 195 previous similar messages [ 9144.086704] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9144.090740] Lustre: Skipped 421 previous similar messages [ 9160.086662] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9160.092849] Lustre: Skipped 847 previous similar messages [ 9192.097692] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9192.101539] Lustre: Skipped 1677 previous similar messages [ 9256.126936] Lustre: *** cfs_fail_loc=a09, val=0*** [ 9256.137217] Lustre: Skipped 2881 previous similar messages [ 9661.602455] Lustre: *** cfs_fail_loc=a08, val=0*** [ 9661.604104] Lustre: Skipped 1833 previous similar messages [ 9661.641335] Lustre: *** cfs_fail_loc=a08, val=0*** [ 9920.994888] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9921.026929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9921.037473] Lustre: Skipped 1 previous similar message [ 9924.740494] Lustre: server umount lustre-MDT0000 complete [ 9929.215124] LustreError: 169577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850986 with bad export cookie 6482986910230756115 [ 9929.237565] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9929.679599] Lustre: server umount lustre-OST0000 complete [ 9934.754242] Lustre: server umount lustre-OST0001 complete [ 9942.657196] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_hostid [ 9951.533172] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [ 9961.347111] vdc: vdc1 vdc9 [ 9970.662229] vde: vde1 vde9 [ 9980.499593] vdf: vdf1 vdf9 [ 9997.198460] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [10006.141922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10006.480854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10006.548402] Lustre: lustre-MDT0000: new disk, initializing [10006.821170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10006.886618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10011.529329] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10016.463704] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10023.314613] Lustre: lustre-OST0000: new disk, initializing [10023.317971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10023.321997] Lustre: Skipped 1 previous similar message [10023.410418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10024.709217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10024.715607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10024.904761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10029.921922] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10040.612439] Lustre: lustre-OST0001: new disk, initializing [10040.617213] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10040.729854] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10042.612329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [10042.633684] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [10042.832117] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [10047.144174] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10056.635944] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10060.274655] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10097.344905] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 13:19:13 (1787851153) [10104.139522] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 13:19:20 (1787851160) [10152.816523] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 13:20:09 (1787851209) [10221.419649] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 13:21:17 (1787851277) [10253.772102] Lustre: DEBUG MARKER: rename directory return 255 [10304.528531] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 13:22:40 (1787851360) [10335.987781] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 13:23:12 (1787851392) [10381.680128] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 13:23:58 (1787851438) [10492.714914] LustreError: 196956:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [10530.818182] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 13:26:27 (1787851587) [10576.009145] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 13:27:11 (1787851631) [10626.365652] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 13:28:02 (1787851682) [10820.610301] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 13:31:17 (1787851877) [10822.145710] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test [10823.933506] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 13:31:20 (1787851880) [10890.626920] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61 [10892.449888] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 13:32:29 (1787851949) [10925.966341] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63 [10927.694250] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 13:33:04 (1787851984) [10983.183239] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65 [10984.487550] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 13:34:01 (1787852041) [11034.291607] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 13:34:50 (1787852090) [11036.923250] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode [11039.198389] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 13:34:55 (1787852095) [11069.048670] LustreError: 209700:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1 [11126.217863] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 13:36:22 (1787852182) [11156.092721] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [11158.098646] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [11377.780484] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 13:40:34 (1787852434) [11440.419858] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 13:41:37 (1787852497) [11482.891622] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 13:42:19 (1787852539) [11484.814358] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode [11486.798511] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 13:42:23 (1787852543) [11488.579957] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode [11490.883939] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 13:42:27 (1787852547) [11507.592316] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB) [11524.155300] Lustre: DEBUG MARKER: Write... [11527.103993] Lustre: DEBUG MARKER: Write out of block quota ... [11594.590625] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 13:44:10 (1787852650) [11625.289731] Lustre: DEBUG MARKER: set to use default quota [11627.158383] Lustre: DEBUG MARKER: set default quota [11628.906710] Lustre: DEBUG MARKER: get default quota [11635.493534] Lustre: DEBUG MARKER: Test not out of quota [11640.426428] Lustre: DEBUG MARKER: Test out of quota [11653.178807] Lustre: DEBUG MARKER: Increase default quota [11688.667408] Lustre: DEBUG MARKER: Set quota to override default quota [11688.761673] LustreError: 186764:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1788457546 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [11703.211702] Lustre: DEBUG MARKER: Set to use default quota again [11729.010856] Lustre: DEBUG MARKER: Cleanup [11803.563766] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 13:47:39 (1787852859) [11834.720820] Lustre: DEBUG MARKER: set default quota for qpool1 [11838.697821] Lustre: DEBUG MARKER: Write from user that hasn't lqe [11893.676176] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 13:49:10 (1787852950) [11973.155819] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 13:50:29 (1787853029) [12042.909934] Lustre: DEBUG MARKER: Write... [12045.866701] Lustre: DEBUG MARKER: Write out of block quota ... [12170.341562] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 13:53:46 (1787853226) [12219.777903] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 13:54:36 (1787853276) [12226.700492] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 13:54:43 (1787853283) [12228.233323] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate [12230.073628] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 13:54:46 (1787853286) [12231.630378] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate [12233.857151] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 13:54:50 (1787853290) [12253.071773] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 13:55:09 (1787853309) [12254.809923] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode [12256.415787] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 13:55:13 (1787853313) [12272.512606] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [12278.964637] LustreError: 234747:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms [12282.014145] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.25@tcp (stopping) [12284.897077] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12284.898256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12284.916061] Lustre: Skipped 2 previous similar messages [12284.930985] Lustre: Skipped 1 previous similar message [12287.097204] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.25@tcp (stopping) [12289.023114] LustreError: 234747:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake [12289.324929] Lustre: server umount lustre-MDT0000 complete [12296.387649] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12296.913581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12297.138215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65) [12297.139923] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65) [12301.121517] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12304.357144] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12304.364564] Lustre: Skipped 1 previous similar message [12342.155580] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 13:56:38 (1787853398) [12374.387810] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 13:57:11 (1787853431) [12384.500851] LustreError: 3316:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed. [12384.516331] LustreError: 3316:0:(qsd_reint.c:633:qqi_reint_delayed()) Skipped 2 previous similar messages [12405.401207] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 13:57:42 (1787853462) [12457.830746] Lustre: *** cfs_fail_loc=a08, val=0*** [12457.834715] Lustre: Skipped 10 previous similar messages [12457.845933] Lustre: *** cfs_fail_loc=a08, val=0*** [12457.851274] Lustre: Skipped 15 previous similar messages [12566.971763] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 14:00:23 (1787853623) [12644.989396] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 14:01:41 (1787853701) [12720.704750] LustreError: 236171:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788458578 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [12875.236048] LustreError: 235336:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788458732 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [13023.808351] LustreError: 235336:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788458881 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [13132.418754] Lustre: DEBUG MARKER: == sanity-quota test 87: lfs quota -a should print default quota setting ========================================================== 14:09:49 (1787854189) [13143.002352] Lustre: *** cfs_fail_loc=a09, val=0*** [13147.031869] Lustre: *** cfs_fail_loc=a09, val=0*** [13147.035849] Lustre: Skipped 173 previous similar messages [13155.032781] Lustre: *** cfs_fail_loc=a09, val=0*** [13155.035285] Lustre: Skipped 359 previous similar messages [13229.026657] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13229.028939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13229.048432] Lustre: Skipped 1 previous similar message [13231.529512] Lustre: server umount lustre-MDT0000 complete [13239.482245] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13240.085396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13240.251891] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:97) [13240.279249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:68 to 0x240000400:97) [13244.717591] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13248.550959] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13248.562647] Lustre: Skipped 1 previous similar message [13266.832892] Lustre: DEBUG MARKER: == sanity-quota test 88: Writing over quota should not hang ========================================================== 14:12:03 (1787854323) [13268.951402] Lustre: DEBUG MARKER: SKIP: sanity-quota test_88 require client with >4k pages [13271.087378] Lustre: DEBUG MARKER: == sanity-quota test 89: Show default quota with squash_uid ========================================================== 14:12:07 (1787854327) [13294.163643] Lustre: DEBUG MARKER: == sanity-quota test 90a: lfs quota should work without mount point ========================================================== 14:12:30 (1787854350) [13314.221067] Lustre: DEBUG MARKER: == sanity-quota test 90b: lfs quota should work with multiple mount points ========================================================== 14:12:50 (1787854370) [13335.234593] Lustre: DEBUG MARKER: == sanity-quota test 91: new quota index files in quota_master ========================================================== 14:13:11 (1787854391) [13350.880698] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13350.901566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13350.913753] Lustre: Skipped 3 previous similar messages [13352.527366] Lustre: server umount lustre-MDT0000 complete [13355.998948] LustreError: 186750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854413 with bad export cookie 6482986910232258728 [13356.005859] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13356.024899] LustreError: 186750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13356.026957] LustreError: 3317:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [13358.136946] Lustre: server umount lustre-OST0000 complete [13363.903366] Lustre: server umount lustre-OST0001 complete [13370.695791] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_hostid [13378.582656] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [13388.185813] vdc: vdc1 vdc9 [13397.450714] vde: vde1 vde9 [13407.037692] vdf: vdf1 vdf9 [13418.012757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13418.211529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13418.281426] Lustre: lustre-MDT0000: new disk, initializing [13418.756249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13418.825698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13423.517360] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13433.066415] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13439.676937] Lustre: lustre-OST0000: new disk, initializing [13439.690753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13439.707412] Lustre: Skipped 1 previous similar message [13439.791641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13440.976080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13440.987633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13441.184428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13445.976386] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13453.989173] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13460.268311] Lustre: lustre-OST0001: new disk, initializing [13460.271116] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13460.342666] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13461.880422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [13461.893389] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [13462.056132] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [13466.546905] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13476.272542] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13487.249719] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.25@tcp (stopping) [13487.589908] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13487.612819] Lustre: Skipped 2 previous similar messages [13489.012663] Lustre: server umount lustre-MDT0000 complete [13500.059747] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13500.616917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13505.611461] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13508.080561] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13508.088208] Lustre: Skipped 1 previous similar message [13513.081556] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13514.694356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13523.431871] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13523.458439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13523.469258] Lustre: Skipped 3 previous similar messages [13528.545493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13528.551584] Lustre: Skipped 1 previous similar message [13529.774803] Lustre: server umount lustre-MDT0000 complete [13533.494448] LustreError: 252006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854591 with bad export cookie 6482986910232261164 [13533.501214] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13533.504483] LustreError: 252006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13533.569812] Lustre: server umount lustre-OST0000 complete [13537.261770] Lustre: server umount lustre-OST0001 complete [13550.707728] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_hostid [13557.226885] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [13565.396114] vdc: vdc1 vdc9 [13575.114680] vde: vde1 vde9 [13585.174361] vdf: vdf1 vdf9 [13585.219775] vdf: vdf1 vdf9 [13585.257672] vdf: vdf1 vdf9 [13600.933219] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [13612.238252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13612.508435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13612.732604] Lustre: lustre-MDT0000: new disk, initializing [13613.194749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13613.271723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13619.253751] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13624.818556] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13633.352985] Lustre: lustre-OST0000: new disk, initializing [13633.356865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13633.361907] Lustre: Skipped 1 previous similar message [13633.463802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13635.060642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13635.073251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13635.276906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13640.823864] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13651.584482] Lustre: lustre-OST0001: new disk, initializing [13651.590924] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13653.510365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [13653.518452] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [13653.678404] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [13658.119949] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13666.683185] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13669.985348] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [13678.126121] Lustre: DEBUG MARKER: == sanity-quota test 92: Cannot set inode limit with Quota Pools ========================================================== 14:18:54 (1787854734) [13696.753461] Lustre: DEBUG MARKER: == sanity-quota test 93: update projid while client write to OST ========================================================== 14:19:12 (1787854752) [13725.339195] Lustre: *** cfs_fail_loc=170c, val=0*** [13783.793375] Lustre: DEBUG MARKER: == sanity-quota test 94: lfs quota all respects nodemap offset ========================================================== 14:20:40 (1787854840) [13822.437319] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13822.450436] Lustre: Skipped 1 previous similar message [13822.455986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13822.461329] Lustre: Skipped 1 previous similar message [13828.364538] Lustre: server umount lustre-MDT0000 complete [13837.586660] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13837.864524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13837.868282] Lustre: Skipped 1 previous similar message [13837.959524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [13842.168200] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13851.801314] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.25@tcp (stopping) [13851.816568] Lustre: Skipped 3 previous similar messages [13857.305271] Lustre: server umount lustre-MDT0000 complete [13866.130213] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13866.770898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [13871.173287] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13883.363954] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13883.371754] Lustre: Skipped 1 previous similar message [13884.159655] Lustre: DEBUG MARKER: == sanity-quota test 95a: Correct limits for squashed root ========================================================== 14:22:20 (1787854940) [13909.642499] Lustre: DEBUG MARKER: == sanity-quota test 95b: df respects squashed project id ========================================================== 14:22:46 (1787854966) [13960.258269] Lustre: DEBUG MARKER: == sanity-quota test 96: quota grant should be released when big files are deleted ========================================================== 14:23:36 (1787855016) [13963.032919] Lustre: DEBUG MARKER: SKIP: sanity-quota test_96 the test in only needed to run on LDiskFS [13965.519624] Lustre: DEBUG MARKER: == sanity-quota test 97a: LQA control commands =========== 14:23:41 (1787855021) [13994.542555] LustreError: 273022:0:(qmt_lqa.c:79:qmt_lqa_destroy()) lustre-QMT0000: cannot destroy lqa-dt-lqa1: rc = -2 [13994.550402] LustreError: 273022:0:(qmt_lqa.c:84:qmt_lqa_destroy()) lustre-QMT0000: cannot destroy lqa-md-lqa1: rc = -2 [13996.603266] Lustre: DEBUG MARKER: == sanity-quota test 97b: Check LQA internals ============ 14:24:12 (1787855052) [14018.296146] LustreError: 273821:0:(qmt_lqa.c:250:qmt_lqa_insert_range()) lustre-QMT0000: LQA range 15-31 partially overlaps with existing range 10-19: rc = -34 [14018.303233] LustreError: 273821:0:(qmt_lqa.c:669:qmt_lqa_add()) lustre-QMT0000: lqa:lqa1 can't add range 15:31: rc = -17 [14021.787302] LustreError: 274018:0:(qmt_lqa.c:825:qmt_lqa_remove()) lustre-QMT0000: lqa-md-lqa1 cannot remove range 10:19: rc = -2 [14032.615610] Lustre: DEBUG MARKER: adding 50 LQA ranges took 2s [14038.231776] Lustre: DEBUG MARKER: Removing 50 LQA ranges took 2s [14047.026327] Lustre: DEBUG MARKER: == sanity-quota test 97c: LQA lfs quota commands and MDS restart consistency ========================================================== 14:25:03 (1787855103) [14051.529576] Lustre: Failing over lustre-MDT0000 [14051.834688] Lustre: server umount lustre-MDT0000 complete [14060.538738] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14061.020464] Lustre: 275888:0:(scrub.c:697:lustre_index_register()) lustre-MDT0000: the index [0x200000003:0x74:0x0] has registered with 8/8, may be invalid, replace with 4/8 [14061.078828] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14061.096057] Lustre: Skipped 1 previous similar message [14061.332208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14061.337552] Lustre: Skipped 1 previous similar message [14061.434066] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [14061.714159] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [14061.891677] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [14061.942962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:97) [14066.384097] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14066.671200] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14066.687074] Lustre: Skipped 1 previous similar message [14069.215697] Lustre: 3318:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855110/real 1787855110] req@ffffa0fbc40ada40 x1874687368000896/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787855126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [14072.130186] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [14073.311675] Lustre: 3318:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855115/real 1787855115] req@ffffa0fca5432940 x1874687368001408/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787855131 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [14073.347744] Lustre: 3318:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14081.177066] Lustre: DEBUG MARKER: == sanity-quota test 97d: LQA disk persistence using standard commands ========================================================== 14:25:37 (1787855137) [14104.381149] Lustre: Failing over lustre-MDT0000 [14104.843936] Lustre: server umount lustre-MDT0000 complete [14114.171983] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14114.531363] Lustre: 277941:0:(scrub.c:697:lustre_index_register()) lustre-MDT0000: the index [0x200000003:0x88:0x0] has registered with 8/8, may be invalid, replace with 4/8 [14114.555986] LustreError: 277941:0:(qmt_lqa.c:522:qmt_lqa_load_ranges_from_disk()) lustre-QMT0000: Failed to get record from IAM iterator: rc = -2 [14114.594676] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14114.608755] Lustre: Skipped 2 previous similar messages [14114.968307] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [14118.055737] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [14118.176524] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [14118.244648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:129) [14119.918840] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14119.922647] Lustre: Skipped 1 previous similar message [14120.110532] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14123.999954] Lustre: 3316:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855165/real 1787855165] req@ffffa0fca7f28000 x1874687368018304/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787855181 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [14124.048995] Lustre: 3316:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14126.428177] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [14129.125315] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855170/real 1787855170] req@ffffa0fca7f2b0c0 x1874687368018816/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787855186 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [14129.157278] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [14168.756205] Lustre: DEBUG MARKER: == sanity-quota test 97e: LQA add/remove should reject invalid ranges ========================================================== 14:27:04 (1787855224) [14182.846302] LustreError: 280274:0:(qmt_lqa.c:79:qmt_lqa_destroy()) lustre-QMT0000: cannot destroy lqa-dt-lqa1: rc = -2 [14182.859519] LustreError: 280274:0:(qmt_lqa.c:84:qmt_lqa_destroy()) lustre-QMT0000: cannot destroy lqa-md-lqa1: rc = -2 [14184.801799] Lustre: DEBUG MARKER: == sanity-quota test 98: Verify MDT retry on -EDQUOT during mkdir ========================================================== 14:27:21 (1787855241) [14186.057184] Lustre: DEBUG MARKER: SKIP: sanity-quota test_98 needs >= 2 MDTs [14187.725870] Lustre: DEBUG MARKER: == sanity-quota test 300: inode quota with MDT directory migration at 80% limit ========================================================== 14:27:24 (1787855244) [14189.358764] Lustre: DEBUG MARKER: SKIP: sanity-quota test_300 needs >= 2 MDTs [14194.504551] Lustre: DEBUG MARKER: == sanity-quota test complete, duration 13915 sec ======== 14:27:30 (1787855250) [14196.293358] Lustre: DEBUG MARKER: === sanity-quota: start cleanup 14:27:32 (1787855252) === [14199.785753] Lustre: DEBUG MARKER: === sanity-quota: finish cleanup 14:27:36 (1787855256) === [14206.945614] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14206.946879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14206.980712] Lustre: Skipped 1 previous similar message [14207.000641] Lustre: Skipped 4 previous similar messages [14207.263450] Lustre: server umount lustre-MDT0000 complete [14211.358787] LustreError: 273363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855269 with bad export cookie 6482986910232266673 [14211.372831] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14211.382454] LustreError: 273363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14211.513698] Lustre: server umount lustre-OST0000 complete [14215.850034] Lustre: server umount lustre-OST0001 complete [14228.170872] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing unload_modules_local [14231.113691] Key type lgssc unregistered [14231.456506] LNet: 281818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14231.465864] LNetError: 281818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14231.493223] LNet: Removed LNI 192.168.203.125@tcp [14232.332145] Key type .llcrypt unregistered [14232.337101] Key type ._llcrypt unregistered