[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 504260686 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003296] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.006018] kvm-guest: setup PV IPIs [ 0.008765] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.012044] LSM: Security Framework initializing [ 0.013054] Yama: becoming mindful. [ 0.014052] SELinux: Initializing. [ 0.015077] *** VALIDATE selinux *** [ 0.024666] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029798] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032113] *** VALIDATE tmpfs *** [ 0.033000] *** VALIDATE proc *** [ 0.033246] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035000] Spectre V2 : User space: Vulnerable [ 0.035000] Speculative Store Bypass: Vulnerable [ 0.037376] debug: unmapping init [mem 0xffffffff8ce59000-0xffffffff8ce60fff] [ 0.039243] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041022] ... version: 2 [ 0.042014] ... bit width: 48 [ 0.043016] ... generic registers: 4 [ 0.044014] ... value mask: 0000ffffffffffff [ 0.045016] ... max period: 00007fffffffffff [ 0.046020] ... fixed-purpose events: 3 [ 0.047014] ... event mask: 000000070000000f [ 0.048338] rcu: Hierarchical SRCU implementation. [ 0.050472] smp: Bringing up secondary CPUs ... [ 0.051562] x86: Booting SMP configuration: [ 0.052027] .... node #0, CPUs: #1 #2 #3 [ 0.076028] smp: Brought up 1 node, 4 CPUs [ 0.078020] smpboot: Max logical packages: 1 [ 0.079016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.111021] node 0 deferred pages initialised in 29ms [ 0.115163] devtmpfs: initialized [ 0.116196] x86/mm: Memory block size: 128MB [ 0.120348] gcov: version magic: 0x41383552 [ 0.123465] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.128082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.131321] pinctrl core: initialized pinctrl subsystem [ 0.132181] [ 0.132862] ************************************************************* [ 0.136015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.139012] ** ** [ 0.142014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.145012] ** ** [ 0.148013] ** This means that this kernel is built to expose internal ** [ 0.150013] ** IOMMU data structures, which may compromise security on ** [ 0.154015] ** your system. ** [ 0.157014] ** ** [ 0.160015] ** If you see this message and you are not debugging the ** [ 0.163014] ** kernel, report this immediately to your vendor! ** [ 0.166015] ** ** [ 0.169014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170338] ************************************************************* [ 0.173882] NET: Registered protocol family 16 [ 0.176488] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183114] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187654] cpuidle: using governor menu [ 0.190863] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.195055] PCI: Using configuration type 1 for base access [ 0.197120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.210400] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213080] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.217188] cryptd: max_cpu_qlen set to 1000 [ 0.220415] ACPI: Added _OSI(Module Device) [ 0.221012] ACPI: Added _OSI(Processor Device) [ 0.223124] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230789] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240503] ACPI: Interpreter enabled [ 0.242068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243015] ACPI: Using IOAPIC for interrupt routing [ 0.247114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.248396] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256606] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273249] acpiphp: Slot [2] registered [ 0.274112] acpiphp: Slot [5] registered [ 0.276167] acpiphp: Slot [6] registered [ 0.278325] acpiphp: Slot [7] registered [ 0.280356] acpiphp: Slot [8] registered [ 0.282261] acpiphp: Slot [9] registered [ 0.284164] acpiphp: Slot [10] registered [ 0.286128] acpiphp: Slot [3] registered [ 0.288238] acpiphp: Slot [4] registered [ 0.289149] acpiphp: Slot [11] registered [ 0.291103] acpiphp: Slot [12] registered [ 0.292101] acpiphp: Slot [13] registered [ 0.294133] acpiphp: Slot [14] registered [ 0.296100] acpiphp: Slot [15] registered [ 0.298143] acpiphp: Slot [16] registered [ 0.300143] acpiphp: Slot [17] registered [ 0.302113] acpiphp: Slot [18] registered [ 0.304177] acpiphp: Slot [19] registered [ 0.306193] acpiphp: Slot [20] registered [ 0.307173] acpiphp: Slot [21] registered [ 0.310127] acpiphp: Slot [22] registered [ 0.312111] acpiphp: Slot [23] registered [ 0.314265] acpiphp: Slot [24] registered [ 0.315139] acpiphp: Slot [25] registered [ 0.317223] acpiphp: Slot [26] registered [ 0.320126] acpiphp: Slot [27] registered [ 0.322125] acpiphp: Slot [28] registered [ 0.323161] acpiphp: Slot [29] registered [ 0.326876] acpiphp: Slot [30] registered [ 0.328156] acpiphp: Slot [31] registered [ 0.329081] PCI host bridge to bus 0000:00 [ 0.330000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.341028] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.345026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348170] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352308] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.356344] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.366016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.372000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.375022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.381017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.383596] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.386714] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.389048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.392984] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.411020] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.418017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430031] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.444029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.460021] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.467000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.500030] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.516174] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.532025] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.545490] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.556025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.565024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.590026] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.607315] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.618029] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.629029] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.657060] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.669906] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.680030] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.687039] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.710027] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.729864] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.752043] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.762017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.791018] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.808740] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.819467] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.829590] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.842375] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.858314] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.866063] iommu: Default domain type: Passthrough [ 0.872026] SCSI subsystem initialized [ 0.876188] ACPI: bus type USB registered [ 0.882146] usbcore: registered new interface driver usbfs [ 0.886101] usbcore: registered new interface driver hub [ 0.895095] usbcore: registered new device driver usb [ 0.899182] pps_core: LinuxPPS API ver. 1 registered [ 0.902012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.913086] PTP clock support registered [ 0.921111] EDAC MC: Ver: 3.0.0 [ 0.923265] PCI: Using ACPI for IRQ routing [ 0.924827] NetLabel: Initializing [ 0.925015] NetLabel: domain hash size = 128 [ 0.926014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.928000] NetLabel: unlabeled traffic allowed by default [ 0.930065] vgaarb: loaded [ 0.936321] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.938017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.948041] clocksource: Switched to clocksource kvm-clock [ 1.072447] VFS: Disk quotas dquot_6.6.0 [ 1.074140] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.076948] *** VALIDATE ramfs *** [ 1.078729] *** VALIDATE hugetlbfs *** [ 1.081394] pnp: PnP ACPI init [ 1.083933] pnp: PnP ACPI: found 6 devices [ 1.120620] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.124339] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.126777] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.132602] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.137032] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.142597] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.149586] NET: Registered protocol family 2 [ 1.159574] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.166566] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.170392] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.180835] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.186687] TCP: Hash tables configured (established 65536 bind 65536) [ 1.194615] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.202913] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.209859] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.221353] NET: Registered protocol family 1 [ 1.226338] RPC: Registered named UNIX socket transport module. [ 1.230608] RPC: Registered udp transport module. [ 1.234993] RPC: Registered tcp transport module. [ 1.236883] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.245806] NET: Registered protocol family 44 [ 1.248188] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.250696] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.256641] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.260562] PCI: CLS 0 bytes, default 64 [ 1.265595] Unpacking initramfs... [ 4.073678] debug: unmapping init [mem 0xffff9f003cc54000-0xffff9f003ffbffff] [ 4.081697] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.083811] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.086446] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.439318] Initialise system trusted keyrings [ 5.450880] Key type blacklist registered [ 5.460270] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.469956] zbud: loaded [ 5.474742] *** VALIDATE nfs *** [ 5.476160] *** VALIDATE nfs4 *** [ 5.481305] pstore: using deflate compression [ 5.489539] Platform Keyring initialized [ 5.693793] NET: Registered protocol family 38 [ 5.695286] Key type asymmetric registered [ 5.696573] Asymmetric key parser 'x509' registered [ 5.698935] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.703208] io scheduler mq-deadline registered [ 5.704570] io scheduler kyber registered [ 5.706281] io scheduler bfq registered [ 5.708688] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.711926] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.714294] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.720054] ACPI: Power Button [PWRF] [ 5.828907] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.059333] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.702676] hrtimer: interrupt took 3685529 ns [ 6.709995] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.959053] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.436179] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.495367] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.553534] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.561806] Non-volatile memory driver v1.3 [ 7.563403] Linux agpgart interface v0.103 [ 7.613245] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 7.621850] vda: detected capacity change from 0 to 68530176 [ 7.656219] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.662557] vdb: detected capacity change from 0 to 1073741824 [ 7.695924] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.700036] vdc: detected capacity change from 0 to 2621440000 [ 7.726570] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.734591] vdd: detected capacity change from 0 to 2621440000 [ 7.762133] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.765727] vde: detected capacity change from 0 to 4294967296 [ 7.793890] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.797407] vdf: detected capacity change from 0 to 4294967296 [ 7.805938] libphy: Fixed MDIO Bus: probed [ 7.821382] usbcore: registered new interface driver usbserial_generic [ 7.824065] usbserial: USB Serial support registered for generic [ 7.837065] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.847659] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.851876] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.855176] mousedev: PS/2 mouse device common for all mice [ 7.865896] rtc_cmos 00:05: RTC can wake from S4 [ 7.876329] rtc_cmos 00:05: registered as rtc0 [ 7.878939] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.882906] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.889246] intel_pstate: CPU model not supported [ 7.895939] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.897896] hid: raw HID events driver (C) Jiri Kosina [ 7.908510] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.914339] usbcore: registered new interface driver usbhid [ 7.919376] usbhid: USB HID core driver [ 7.921103] drop_monitor: Initializing network drop monitor service [ 7.923620] Initializing XFRM netlink socket [ 7.931064] NET: Registered protocol family 10 [ 7.934917] Segment Routing with IPv6 [ 7.936969] NET: Registered protocol family 17 [ 7.940192] mpls_gso: MPLS GSO support [ 7.945844] RAS: Correctable Errors collector initialized. [ 7.948915] AVX version of gcm_enc/dec engaged. [ 7.953711] AES CTR mode by8 optimization enabled [ 8.109820] sched_clock: Marking stable (8109801058, 0)->(9262675052, -1152873994) [ 8.117706] registered taskstats version 1 [ 8.120883] Loading compiled-in X.509 certificates [ 8.126696] zswap: loaded using pool lzo/zbud [ 8.192160] Key type big_key registered [ 8.212952] Key type encrypted registered [ 8.215912] ima: No TPM chip found, activating TPM-bypass! [ 8.218837] ima: Allocated hash algorithm: sha1 [ 8.221433] ima: No architecture policies found [ 8.223711] evm: Initialising EVM extended attributes: [ 8.226949] evm: security.selinux [ 8.228202] evm: security.ima [ 8.229419] evm: security.capability [ 8.230621] evm: HMAC attrs: 0x1 [ 8.236147] rtc_cmos 00:05: setting system clock to 2025-11-17 00:24:32 UTC (1763339072) [ 8.244201] debug: unmapping init [mem 0xffffffff8de03000-0xffffffff8dffffff] [ 8.247993] debug: unmapping init [mem 0xffffffff8cb82000-0xffffffff8ce58fff] [ 8.258165] Write protecting the kernel read-only data: 28672k [ 8.265572] debug: unmapping init [mem 0xffffffff8b203000-0xffffffff8b3fffff] [ 8.272456] debug: unmapping init [mem 0xffffffff8bb14000-0xffffffff8bbfffff] [ 8.356578] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.366819] systemd[1]: Detected virtualization kvm. [ 8.368915] systemd[1]: Detected architecture x86-64. [ 8.371180] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.418778] systemd[1]: No hostname configured. [ 8.420591] systemd[1]: Set hostname to . [ 8.429501] random: systemd: uninitialized urandom read (16 bytes read) [ 8.445212] systemd[1]: Initializing machine ID from random generator. [ 8.537820] random: ln: uninitialized urandom read (6 bytes read) [ 8.684510] random: systemd: uninitialized urandom read (16 bytes read) [ 8.688602] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 8.695574] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 8.700719] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.998815] device-mapper: uevent: version 1.0.3 [ 10.001895] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 11.709794] random: fast init done [ 11.905119] virtio_net virtio0 ens2: renamed from eth0 [ 12.247433] scsi host0: ata_piix [ 12.472088] scsi host1: ata_piix [ 12.478489] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.484312] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.753558] random: crng init done [ 16.755089] random: 7 urandom warning(s) missed due to ratelimiting [ 17.707028] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.397292] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.238205] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.164995] SELinux: Disabled at runtime. [ 23.251160] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.262230] systemd[1]: Detected virtualization kvm. [ 23.264686] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.777118] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.781926] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.794293] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.806433] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.813504] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.828138] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.849374] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ 26.228545] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 27.060404] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.141728] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.177385] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.627853] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.729938] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 34.418452] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.214796] NFS: Registering the id_resolver key type [ 35.217100] Key type id_resolver registered [ 35.218811] Key type id_legacy registered [ ***] A start job is running for Configur…-only root support (9s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg108-server login: [ 87.357435] libcfs: loading out-of-tree module taints kernel. [ 87.382975] Key type ._llcrypt registered [ 87.384359] Key type .llcrypt registered [ 87.471303] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 99.534913] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 100.798173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 100.820682] alg: No test for adler32 (adler32-zlib) [ 102.094879] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 103.052600] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 104.864198] Key type lgssc registered [ 106.767945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.222114] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 121.740100] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 127.894371] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.899225] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 140.220065] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.970833] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 162.950121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 163.018388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 163.039766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.233365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.265610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 164.337940] Lustre: lustre-MDT0000: new disk, initializing [ 164.417882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.435896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 167.733147] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.315034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 178.399102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 178.501468] Lustre: 6493:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 178.544371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 178.547400] Lustre: Skipped 1 previous similar message [ 178.633919] Lustre: lustre-MDT0001: new disk, initializing [ 178.740850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.780085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.788308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.596099] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.332757] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 195.160445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 195.251510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 195.461583] Lustre: lustre-OST0000: new disk, initializing [ 195.464334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 195.519640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.319931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 196.325394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.375177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 200.318519] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.288872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 211.356332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 211.422347] Lustre: lustre-OST0001: new disk, initializing [ 211.424864] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 211.476628] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.287344] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.170352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 220.175241] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 220.259879] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 224.264533] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 229.507190] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 238.408934] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing check_logdir /tmp/testlogs/ [ 241.101213] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing yml_node [ 244.405757] Lustre: DEBUG MARKER: Client: 2.16.61.49 [ 246.565441] Lustre: DEBUG MARKER: MDS: 2.16.61.49 [ 248.461054] Lustre: DEBUG MARKER: OSS: 2.16.61.49 [ 249.664057] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-quota ============----- Sun Nov 16 19:28:33 EST 2025 [ 261.839684] Lustre: DEBUG MARKER: excepting tests: 2 4a 63 65 [ 262.785932] Lustre: DEBUG MARKER: skipping tests SLOW=no: 61 [ 264.492667] Lustre: DEBUG MARKER: === sanity-quota: start setup 19:28:48 (1763339328) === [ 267.108195] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing check_config_client /mnt/lustre [ 280.128456] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 282.462993] Lustre: 13267:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 284.933055] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 288.835857] Lustre: DEBUG MARKER: === sanity-quota: finish setup 19:29:12 (1763339352) === [ 328.187235] Lustre: DEBUG MARKER: == sanity-quota test 0: Test basic quota performance ===== 19:29:51 (1763339391) [ 367.614159] Lustre: DEBUG MARKER: == sanity-quota test 1a: Block hard limit (normal use and out of quota) ========================================================== 19:30:30 (1763339430) [ 377.032862] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 382.236712] Lustre: DEBUG MARKER: Write... [ 384.173977] Lustre: DEBUG MARKER: Write out of block quota ... [ 411.980734] Lustre: DEBUG MARKER: -------------------------------------- [ 413.651563] Lustre: DEBUG MARKER: Group quota (block hardlimit:10 MB) [ 418.564761] Lustre: DEBUG MARKER: Write... [ 420.682933] Lustre: DEBUG MARKER: Write out of block quota ... [ 449.317991] Lustre: DEBUG MARKER: -------------------------------------- [ 450.800941] Lustre: DEBUG MARKER: Project quota (block hardlimit:10 mb) [ 453.006652] Lustre: DEBUG MARKER: Write... [ 454.783617] Lustre: DEBUG MARKER: Write out of block quota ... [ 494.450364] Lustre: DEBUG MARKER: == sanity-quota test 1b: Quota pools: Block hard limit (normal use and out of quota) ========================================================== 19:32:37 (1763339557) [ 504.069862] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 519.668902] Lustre: DEBUG MARKER: Write... [ 522.151983] Lustre: DEBUG MARKER: Write out of block quota ... [ 548.058567] Lustre: DEBUG MARKER: -------------------------------------- [ 549.414474] Lustre: DEBUG MARKER: Group quota (block hardlimit:20 MB) [ 554.270468] Lustre: DEBUG MARKER: Write... [ 556.397675] Lustre: DEBUG MARKER: Write out of block quota ... [ 586.222917] Lustre: DEBUG MARKER: -------------------------------------- [ 587.719624] Lustre: DEBUG MARKER: Project quota (block hardlimit:20 mb) [ 590.685991] Lustre: DEBUG MARKER: Write... [ 592.824510] Lustre: DEBUG MARKER: Write out of block quota ... [ 641.440459] Lustre: DEBUG MARKER: == sanity-quota test 1c: Quota pools: check 3 pools with hardlimit only for global ========================================================== 19:35:04 (1763339704) [ 651.599393] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 677.263644] Lustre: DEBUG MARKER: Write... [ 680.141517] Lustre: DEBUG MARKER: Write out of block quota ... [ 759.977220] Lustre: DEBUG MARKER: == sanity-quota test 1d: Quota pools: check block hardlimit on different pools ========================================================== 19:37:03 (1763339823) [ 769.696871] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 792.462662] Lustre: DEBUG MARKER: Write... [ 794.579707] Lustre: DEBUG MARKER: Write out of block quota ... [ 866.307254] Lustre: DEBUG MARKER: == sanity-quota test 1e: Quota pools: global pool high block limit vs quota pool with small ========================================================== 19:38:49 (1763339929) [ 876.383169] Lustre: DEBUG MARKER: User quota (block hardlimit:53000000 MB) [ 887.344760] Lustre: DEBUG MARKER: Write... [ 889.281169] Lustre: DEBUG MARKER: Write out of block quota ... [ 899.247650] Lustre: DEBUG MARKER: Write... [ 939.723588] Lustre: DEBUG MARKER: == sanity-quota test 1f: Quota pools: correct qunit after removing/adding OST ========================================================== 19:40:03 (1763340003) [ 949.188501] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 961.450460] Lustre: DEBUG MARKER: Write... [ 963.191937] Lustre: DEBUG MARKER: Write out of block quota ... [ 997.381247] Lustre: DEBUG MARKER: Write... [ 999.252807] Lustre: DEBUG MARKER: Write out of block quota ... [ 1042.649507] Lustre: DEBUG MARKER: == sanity-quota test 1g: Quota pools: Block hard limit with wide striping ========================================================== 19:41:46 (1763340106) [ 1052.603718] Lustre: DEBUG MARKER: User quota (block hardlimit:40 MB) [ 1063.593163] Lustre: 6500:0:(osd_handler.c:2183:osd_trans_start()) lustre-MDT0000: credits 6546 > trans_max 3200 [ 1063.597341] Lustre: 6500:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 4/16/0, destroy: 0/0/0 [ 1063.601136] Lustre: 6500:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 401/401/0, xattr_set: 602/5615/0 [ 1063.607375] Lustre: 6500:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 20/118/0, punch: 0/0/0, quota 8/328/0 [ 1063.611706] Lustre: 6500:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/68/0, delete: 0/0/0 [ 1063.615940] Lustre: 6500:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1063.621234] CPU: 2 PID: 6500 Comm: mdt00_001 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1063.625921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 1063.629039] Call Trace: [ 1063.630003] ? dump_stack+0xbb/0x10e [ 1063.631601] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [ 1063.633685] ? top_trans_start+0x599/0xd80 [ptlrpc] [ 1063.636037] ? lod_ref_add+0x30/0x30 [lod] [ 1063.637708] ? lod_trans_start+0x109/0x4c0 [lod] [ 1063.639627] ? mdd_declare_attr_set+0x190/0x690 [mdd] [ 1063.641683] ? mdd_env_info+0x25/0xc0 [mdd] [ 1063.643437] ? mdd_trans_start+0x18/0x30 [mdd] [ 1063.645836] ? mdd_attr_set+0xa5a/0x1250 [mdd] [ 1063.647988] ? mdt_reint_setattr+0x1342/0x1f90 [mdt] [ 1063.650104] ? mdt_reint_setattr+0x1342/0x1f90 [mdt] [ 1063.652030] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 1063.653494] ? mdt_reint_internal+0x6a0/0xdc0 [mdt] [ 1063.655588] ? mdt_reint+0x163/0x190 [mdt] [ 1063.657310] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 1063.659670] ? tgt_request_handle+0x573/0x1e70 [ptlrpc] [ 1063.661909] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 1063.665382] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 1063.668411] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 1063.670765] ? ptlrpc_wait_event+0x980/0x980 [ptlrpc] [ 1063.673331] ? kthread+0x1d1/0x200 [ 1063.674464] ? set_kthread_struct+0x70/0x70 [ 1063.675912] ? ret_from_fork+0x1f/0x30 [ 1065.092246] Lustre: DEBUG MARKER: Write... [ 1076.922786] Lustre: DEBUG MARKER: Write out of block quota ... [ 1133.035832] Lustre: DEBUG MARKER: == sanity-quota test 1h: Block hard limit test using fallocate ========================================================== 19:43:16 (1763340196) [ 1143.610117] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 1148.169619] Lustre: DEBUG MARKER: Write 5MiB Using Fallocate [ 1153.884418] Lustre: DEBUG MARKER: Write 11MiB Using Fallocate [ 1210.858565] Lustre: DEBUG MARKER: == sanity-quota test 1i: Quota pools: different limit and usage relations ========================================================== 19:44:34 (1763340274) [ 1218.875568] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 1228.972326] Lustre: DEBUG MARKER: Write... [ 1230.433326] Lustre: DEBUG MARKER: Write out of block quota ... [ 1255.935654] Lustre: DEBUG MARKER: Write... [ 1257.718206] Lustre: DEBUG MARKER: Write out of block quota ... [ 1267.798919] LustreError: 6501:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:14344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 1299.054894] Lustre: DEBUG MARKER: == sanity-quota test 1j: Enable project quota enforcement for root ========================================================== 19:46:02 (1763340362) [ 1311.361262] Lustre: DEBUG MARKER: -------------------------------------- [ 1312.230605] Lustre: DEBUG MARKER: Project quota (hardlimit: 40 mb, 2048 files) [ 1668.845831] Lustre: DEBUG MARKER: SKIP: sanity-quota test_2 skipping excluded test 2 [ 1670.139185] Lustre: DEBUG MARKER: == sanity-quota test 3a: Block soft limit (start timer, timer goes off, stop timer) ========================================================== 19:52:13 (1763340733) [ 1710.145481] Lustre: DEBUG MARKER: Write after timer goes off [ 1713.310732] Lustre: DEBUG MARKER: Write after cancel lru locks [ 1772.630446] Lustre: DEBUG MARKER: Write after timer goes off [ 1773.996970] Lustre: DEBUG MARKER: Write after cancel lru locks [ 1840.589036] Lustre: DEBUG MARKER: Write after timer goes off [ 1844.257786] Lustre: DEBUG MARKER: Write after cancel lru locks [ 1896.881676] Lustre: DEBUG MARKER: == sanity-quota test 3b: Quota pools: Block soft limit (start timer, expires, stop timer) ========================================================== 19:56:00 (1763340960) [ 1942.522787] Lustre: DEBUG MARKER: Write after timer goes off [ 1946.077582] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2009.721958] Lustre: DEBUG MARKER: Write after timer goes off [ 2013.452400] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2078.948390] Lustre: DEBUG MARKER: Write after timer goes off [ 2083.844052] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2151.192909] Lustre: DEBUG MARKER: == sanity-quota test 3c: Quota pools: check block soft limit on different pools ========================================================== 20:00:14 (1763341214) [ 2206.589160] Lustre: DEBUG MARKER: Write after timer goes off [ 2208.172466] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2278.135186] Lustre: DEBUG MARKER: SKIP: sanity-quota test_4a skipping excluded test 4a [ 2279.722846] Lustre: DEBUG MARKER: == sanity-quota test 4b: Grace time strings handling ===== 20:02:23 (1763341343) [ 2288.753570] Lustre: DEBUG MARKER: == sanity-quota test 5: Chown [ 2407.386185] Lustre: DEBUG MARKER: == sanity-quota test 6: Test dropping acquire request on master ========================================================== 20:04:30 (1763341470) [ 2442.208723] Lustre: *** cfs_fail_loc=513, val=601*** [ 2442.217978] Lustre: Skipped 12 previous similar messages [ 2443.232698] Lustre: *** cfs_fail_loc=513, val=601*** [ 2443.238740] Lustre: Skipped 4 previous similar messages [ 2444.257458] Lustre: *** cfs_fail_loc=513, val=601*** [ 2444.259743] Lustre: Skipped 6 previous similar messages [ 2444.737996] LustreError: 15810:0:(service.c:2319:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1848995133055872 [ 2446.818551] Lustre: *** cfs_fail_loc=513, val=601*** [ 2446.825770] Lustre: Skipped 13 previous similar messages [ 2451.936929] Lustre: *** cfs_fail_loc=513, val=601*** [ 2451.939124] Lustre: Skipped 21 previous similar messages [ 2459.104849] Lustre: 8389:0:(service.c:1607:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff9eff82ba1500 x1848995123822208/t0(0) o4->498fc4a5-303b-4b94-b964-fac57cff59c9@192.168.201.8@tcp:523/0 lens 488/448 e 1 to 0 dl 1763341528 ref 2 fl Interpret:/600/0 rc 0/0 job:'dd.60000' uid:60000 gid:60000 projid:0 [ 2461.152185] Lustre: 39332:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763341509/real 1763341509] req@ffff9eff8feb3800 x1848995133055872/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1763341525 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_006.0' uid:0 gid:0 projid:4294967295 [ 2461.186705] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.197199] Lustre: *** cfs_fail_loc=513, val=601*** [ 2461.198917] Lustre: Skipped 40 previous similar messages [ 2461.207455] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 2461.227761] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2461.379398] LustreError: 15810:0:(service.c:2319:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1848995133064320 [ 2477.536245] Lustre: 3638:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763341525/real 1763341525] req@ffff9eff8d738380 x1848995133064320/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1763341541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_000.0' uid:0 gid:0 projid:4294967295 [ 2477.539492] Lustre: *** cfs_fail_loc=513, val=601*** [ 2477.594094] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.598468] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 2477.604069] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2477.605612] LustreError: 15810:0:(service.c:2319:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1848995133073408 [ 2477.609454] Lustre: Skipped 93 previous similar messages [ 2492.896182] Lustre: 39332:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763341541/real 1763341541] req@ffff9eff813d1880 x1848995133073408/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1763341557 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_006.0' uid:0 gid:0 projid:4294967295 [ 2492.926076] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.949848] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 2492.965735] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2523.029172] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 20:06:26 (1763341586) [ 2547.178041] Lustre: Failing over lustre-OST0000 [ 2549.220709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.223958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.241913] Lustre: Skipped 1 previous similar message [ 2549.268129] LustreError: 73684:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2549.336817] Lustre: server umount lustre-OST0000 complete [ 2555.361469] LustreError: 8386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.368700] LustreError: 8386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2559.462631] LustreError: 39418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.473420] LustreError: 39418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2560.473378] LustreError: 39418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2560.539545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2560.807393] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 2560.853657] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2561.997161] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2562.837367] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2562.839711] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2566.482157] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2574.778696] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2580.249444] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2585.828513] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2591.181497] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2595.877673] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2600.073941] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2620.608920] Lustre: Failing over lustre-OST0000 [ 2620.654098] LustreError: 77146:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2620.656386] LustreError: 77146:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2620.713627] Lustre: server umount lustre-OST0000 complete [ 2621.932893] LustreError: 11992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.434852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.471836] Lustre: Skipped 1 previous similar message [ 2627.029164] LustreError: 11992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.073755] LustreError: 11992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2628.998360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2629.318298] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 2629.339338] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2630.535058] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2631.318641] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2631.322070] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2631.334945] Lustre: Skipped 1 previous similar message [ 2634.500454] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2641.714499] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2646.344337] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2650.980235] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2655.415944] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2660.101080] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2664.934553] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2687.337319] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) ========================================================== 20:09:10 (1763341750) [ 2711.735748] Lustre: *** cfs_fail_loc=a02, val=0*** [ 2717.840346] LustreError: 3636:0:(qsd_handler.c:337:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8 qsd:lustre-OST0000 qtype:usr id:60000 enforced:1 granted: 1024 pending:0 waiting:0 req:1 usage: 2048 qunit:0 qtune:0 edquot:0 default:no revoke:0 truncated:0 [ 2717.880310] Lustre: Failing over lustre-OST0000 [ 2717.910136] LustreError: 82029:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2717.924496] LustreError: 82029:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2717.984749] Lustre: server umount lustre-OST0000 complete [ 2718.690293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2718.690883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.694039] LustreError: Skipped 1 previous similar message [ 2718.704558] LustreError: 39418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.718196] LustreError: 39418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2724.772935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2725.071655] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 2725.089933] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2726.483231] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2726.758632] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2726.761296] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2726.767230] Lustre: Skipped 1 previous similar message [ 2729.386353] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2734.011375] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2737.045269] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2740.163145] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2743.177163] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2746.692613] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2750.169137] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2778.916804] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ========================================================== 20:10:42 (1763341842) [ 2798.270182] LustreError: 86802:0:(qsd_reint.c:475:qsd_reint_main()) cfs_fail_timeout id a03 sleeping for 10000ms [ 2798.281354] LustreError: 86802:0:(qsd_reint.c:475:qsd_reint_main()) Skipped 5 previous similar messages [ 2799.951630] Lustre: Failing over lustre-MDT0000 [ 2800.251879] LustreError: 86905:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2800.256828] LustreError: 86905:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2800.451174] Lustre: server umount lustre-MDT0000 complete [ 2800.616851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2800.630437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.639404] Lustre: Skipped 1 previous similar message [ 2800.643291] LustreError: 19470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.650985] LustreError: 19470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2803.677687] LustreError: 86805:0:(qsd_reint.c:475:qsd_reint_main()) cfs_fail_timeout interrupted [ 2809.425372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.543689] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2809.815536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2809.882929] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2811.363663] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2813.065161] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2814.946514] LustreError: 3634:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x2020000:0x0].0x0 (ffff9eff82cd9e00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2814.974996] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2814.985360] Lustre: Skipped 1 previous similar message [ 2815.054019] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2815.055948] Lustre: 86807:0:(qsd_reint.c:241:qsd_reint_index()) lustre-OST0001: index version for fid [0x200000005:0x100c:0x0] is 0, but index isn't empty (1) [ 2815.113334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:144 to 0x280000401:161) [ 2815.114190] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:115 to 0x2c0000401:161) [ 2819.391940] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2916.281763] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2920.241672] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2924.427752] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2928.522341] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2932.245843] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2950.824276] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ========================================================== 20:13:34 (1763342014) [ 2970.020808] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2973.791922] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 3002.737856] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================================== 20:14:26 (1763342066) [ 3022.151983] Lustre: Failing over lustre-MDT0001 [ 3022.260612] LustreError: 96274:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3022.263605] LustreError: 96274:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3022.333130] Lustre: server umount lustre-MDT0001 complete [ 3024.871547] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3024.874743] LustreError: 7151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.884968] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3024.890469] Lustre: Skipped 5 previous similar messages [ 3024.904690] LustreError: 7151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 3032.418184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3032.767736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3032.809663] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 3033.837960] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3035.470442] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3038.190227] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3038.194582] Lustre: Skipped 3 previous similar messages [ 3038.214682] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3038.246698] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 3040.048408] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3043.509527] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 3047.085343] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3050.882636] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 3054.877465] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3058.981413] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 3134.278641] Lustre: Failing over lustre-MDT0001 [ 3134.481033] LustreError: 99442:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3134.484252] LustreError: 99442:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3134.592879] Lustre: server umount lustre-MDT0001 complete [ 3135.460227] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.462430] LustreError: 47577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3135.463181] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3135.471997] Lustre: Skipped 2 previous similar messages [ 3135.487920] LustreError: 47577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3141.350349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3141.674389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3141.721026] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 3143.286904] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3144.622645] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3146.730833] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3146.733563] Lustre: Skipped 2 previous similar messages [ 3146.753553] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3146.781091] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 3150.428132] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3154.452146] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 3158.100384] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3161.881461] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 3165.351306] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3169.934127] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 3250.731628] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ==== 20:18:33 (1763342313) [ 3437.868225] Lustre: DEBUG MARKER: == sanity-quota test 9: Block limit larger than 4GB (b10707) ========================================================== 20:21:41 (1763342501) [ 3439.356824] Lustre: DEBUG MARKER: OST0_SIZE: 3600504 required: 4900000 [ 3445.389596] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ======== 20:21:48 (1763342508) [ 3478.665317] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota ======= 20:22:21 (1763342541) [ 3513.559481] Lustre: DEBUG MARKER: == sanity-quota test 12a: Block quota rebalancing ======== 20:22:57 (1763342577) [ 3569.912143] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ======== 20:23:53 (1763342633) [ 3696.073312] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================================== 20:25:59 (1763342759) [ 3741.143311] Lustre: DEBUG MARKER: == sanity-quota test 14: check panic in qmt_site_recalc_cb ========================================================== 20:26:44 (1763342804) [ 3761.000099] Lustre: Failing over lustre-OST0000 [ 3761.070402] LustreError: 113205:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3761.075686] LustreError: 113205:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3761.130548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3761.142913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3761.174497] LustreError: 11992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3761.194740] LustreError: 11992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3761.266552] Lustre: server umount lustre-OST0000 complete [ 3774.562307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3774.891644] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 3774.943755] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3776.364133] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3776.814484] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3776.815266] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3776.824646] Lustre: Skipped 2 previous similar messages [ 3779.529518] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3809.369630] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ========= 20:27:52 (1763342872) [ 3829.980943] Lustre: DEBUG MARKER: == sanity-quota test 16a: lfs quota should skip the inactive MDT/OST ========================================================== 20:28:13 (1763342893) [ 3847.493926] Lustre: lustre-MDT0001: Client 498fc4a5-303b-4b94-b964-fac57cff59c9 (at 192.168.201.8@tcp) reconnecting [ 3866.172627] Lustre: DEBUG MARKER: == sanity-quota test 16b: lfs quota should skip the nonexistent MDT/OST ========================================================== 20:28:49 (1763342929) [ 3867.502969] Lustre: DEBUG MARKER: SKIP: sanity-quota test_16b needs >= 3 MDTs [ 3868.917446] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error == 20:28:52 (1763342932) [ 3885.662216] Lustre: *** cfs_fail_loc=a04, val=37*** [ 3885.665059] LustreError: 103270:0:(qsd_handler.c:337:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x1 qsd:lustre-OST0001 qtype:usr id:60000 enforced:1 granted: 0 pending:0 waiting:1056 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 truncated:0 [ 3886.791805] Lustre: *** cfs_fail_loc=a04, val=37*** [ 3886.797045] LustreError: 103270:0:(qsd_handler.c:337:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x1 qsd:lustre-OST0001 qtype:usr id:60000 enforced:1 granted: 0 pending:0 waiting:1056 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 truncated:0 [ 3888.651755] Lustre: *** cfs_fail_loc=a04, val=37*** [ 3888.661894] LustreError: 103270:0:(qsd_handler.c:337:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x1 qsd:lustre-OST0001 qtype:usr id:60000 enforced:1 granted: 0 pending:0 waiting:1056 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 truncated:0 [ 3934.128036] Lustre: *** cfs_fail_loc=a04, val=11*** [ 3976.861681] Lustre: *** cfs_fail_loc=a04, val=110*** [ 3976.868097] Lustre: Skipped 1 previous similar message [ 4017.227295] Lustre: *** cfs_fail_loc=a04, val=107*** [ 4017.229160] Lustre: Skipped 1 previous similar message [ 4066.514569] Lustre: DEBUG MARKER: == sanity-quota test 18: MDS failover while writing, no watchdog triggered (b14840) ========================================================== 20:32:10 (1763343130) [ 4075.369426] Lustre: DEBUG MARKER: User quota (limit: 200) [ 4079.887198] Lustre: DEBUG MARKER: Write 100M (buffered) ... [ 4085.751482] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4086.917657] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 4088.247402] Lustre: Failing over lustre-MDT0000 [ 4088.530047] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 4088.533527] Lustre: Skipped 1 previous similar message [ 4088.585663] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107 [ 4088.590671] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.601416] Lustre: Skipped 1 previous similar message [ 4088.608926] LustreError: 125833:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4088.611590] LustreError: 125833:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4088.694876] Lustre: server umount lustre-MDT0000 complete [ 4088.800844] LustreError: 6504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.809099] LustreError: 6504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4108.244918] Lustre: 3637:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763343156/real 1763343156] req@ffff9eff8b8e9c00 x1848995135559040/t0(0) o400->MGC192.168.201.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763343172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4108.263366] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4111.200944] LDISKFS-fs (dm-0): 7 truncates cleaned up [ 4111.203240] LDISKFS-fs (dm-0): recovery complete [ 4111.224131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4118.698289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.735493] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4120.020531] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4121.626634] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4124.132706] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4124.138679] Lustre: Skipped 1 previous similar message [ 4124.194590] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4124.224152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:720 to 0x280000401:737) [ 4124.230851] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:716 to 0x2c0000401:737) [ 4128.051831] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4129.582045] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4136.931938] Lustre: DEBUG MARKER: (dd_pid=108651, time=1, timeout=600) [ 4165.334883] Lustre: DEBUG MARKER: User quota (limit: 200) [ 4169.198640] Lustre: DEBUG MARKER: Write 100M (directio) ... [ 4176.307187] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4177.574912] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 4179.182456] Lustre: Failing over lustre-MDT0000 [ 4179.315966] LustreError: 128878:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4179.322307] LustreError: 128878:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4179.394579] Lustre: server umount lustre-MDT0000 complete [ 4195.808143] Lustre: 3638:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763343244/real 1763343244] req@ffff9eff8c959500 x1848995135618048/t0(0) o400->MGC192.168.201.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763343260 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4195.827259] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4199.521397] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4199.523870] LDISKFS-fs (dm-0): recovery complete [ 4199.530291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.218262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.245741] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4208.084081] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4209.422639] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4211.687839] LustreError: 3634:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff9f009c7bb300) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4211.696138] LustreError: 3634:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 5 previous similar messages [ 4211.728253] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4211.777445] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:716 to 0x2c0000401:769) [ 4211.777963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:769) [ 4216.274947] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4217.625504] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4222.076727] Lustre: DEBUG MARKER: (dd_pid=111008, time=0, timeout=600) [ 4251.780358] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 20:35:15 (1763343315) [ 4290.786250] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 20:35:54 (1763343354) [ 4306.405497] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [ 4315.503673] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [ 4316.742494] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [ 4317.990981] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000 [ 4319.485051] Lustre: DEBUG MARKER: Set quota for 1 times [ 4322.169727] Lustre: DEBUG MARKER: Set quota for 2 times [ 4325.265922] Lustre: DEBUG MARKER: Set quota for 3 times [ 4328.382281] Lustre: DEBUG MARKER: Set quota for 4 times [ 4331.246940] Lustre: DEBUG MARKER: Set quota for 5 times [ 4333.979537] Lustre: DEBUG MARKER: Set quota for 6 times [ 4336.977363] Lustre: DEBUG MARKER: Set quota for 7 times [ 4339.453674] Lustre: DEBUG MARKER: Set quota for 8 times [ 4341.578983] Lustre: DEBUG MARKER: Set quota for 9 times [ 4343.776483] Lustre: DEBUG MARKER: Set quota for 10 times [ 4346.168870] Lustre: DEBUG MARKER: Set quota for 11 times [ 4378.447943] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 20:37:21 (1763343441) [ 4390.887185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4390.887673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4390.891233] LustreError: Skipped 1 previous similar message [ 4390.897298] Lustre: Skipped 4 previous similar messages [ 4395.601284] LustreError: 136125:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4395.603825] LustreError: 136125:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4395.676690] Lustre: server umount lustre-MDT0000 complete [ 4398.824729] LustreError: 103281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343463 with bad export cookie 16511962666445830338 [ 4398.825570] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4398.831081] LustreError: 103281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4399.166090] Lustre: server umount lustre-MDT0001 complete [ 4408.447862] Lustre: server umount lustre-OST0000 complete [ 4411.726450] Lustre: server umount lustre-OST0001 complete [ 4423.919435] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4433.127736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.560048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.742661] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4444.722827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4444.987600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4448.100350] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4450.203348] Lustre: 139020:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4455.251604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4455.539511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4457.779785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 4460.387719] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4468.421765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4468.617165] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4470.542048] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:772 to 0x2c0000401:801) [ 4470.547045] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97) [ 4472.554842] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4478.491947] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4480.504407] Lustre: 140874:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4504.547362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4504.559791] Lustre: Skipped 3 previous similar messages [ 4508.381135] LustreError: 141727:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4508.391600] LustreError: 141727:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4508.498853] Lustre: server umount lustre-MDT0000 complete [ 4511.734099] LustreError: 141428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343575 with bad export cookie 16511962666445838437 [ 4511.737755] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4511.744626] LustreError: 141428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4512.025510] Lustre: server umount lustre-MDT0001 complete [ 4525.503067] Lustre: server umount lustre-OST0000 complete [ 4537.934129] Lustre: server umount lustre-OST0001 complete [ 4549.143399] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4558.421440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.927704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4562.723206] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4570.574867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4573.376627] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4575.368402] Lustre: 144612:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4579.943181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4584.980046] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4585.515992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 4593.140614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4593.605977] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4593.614506] Lustre: Skipped 2 previous similar messages [ 4595.010898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:772 to 0x2c0000401:833) [ 4595.018457] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:129) [ 4598.484547] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4605.933278] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4610.326982] Lustre: 146458:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4625.759920] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 20:41:28 (1763343688) [ 4627.609692] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144 [ 4629.225827] Lustre: DEBUG MARKER: run for 4MB test file [ 4639.236182] Lustre: DEBUG MARKER: User quota (limit: 4 MB) [ 4643.240991] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT [ 4644.558818] Lustre: DEBUG MARKER: Write half of file [ 4646.174092] Lustre: DEBUG MARKER: Write out of block quota ... [ 4647.685441] Lustre: DEBUG MARKER: Step1: done [ 4648.708629] Lustre: DEBUG MARKER: Step2: rewrite should succeed [ 4649.923649] Lustre: DEBUG MARKER: Step2: done [ 4671.407991] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440 [ 4672.919730] Lustre: DEBUG MARKER: run for 40MB test file [ 4682.974461] Lustre: DEBUG MARKER: User quota (limit: 40 MB) [ 4686.805679] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT [ 4688.342729] Lustre: DEBUG MARKER: Write half of file [ 4691.597486] Lustre: DEBUG MARKER: Write out of block quota ... [ 4694.638685] Lustre: DEBUG MARKER: Step1: done [ 4696.408854] Lustre: DEBUG MARKER: Step2: rewrite should succeed [ 4697.871214] Lustre: DEBUG MARKER: Step2: done [ 4734.133617] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 20:43:17 (1763343797) [ 4771.704951] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 20:43:55 (1763343835) [ 4806.681112] Lustre: DEBUG MARKER: Write... [ 4808.851739] Lustre: DEBUG MARKER: Write out of block quota ... [ 4853.826156] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 20:45:17 (1763343917) [ 4859.684829] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 20:45:23 (1763343923) [ 4867.635790] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 20:45:31 (1763343931) [ 4876.222229] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 20:45:39 (1763343939) [ 4885.623179] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 20:45:48 (1763343948) [ 4929.233360] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user [ 5026.367460] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user [ 5120.034832] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 20:49:43 (1763344183) [ 5151.839422] Lustre: DEBUG MARKER: Restart... [ 5154.788506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5154.792982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5154.802803] Lustre: Skipped 17 previous similar messages [ 5154.805729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5156.835459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5156.846078] Lustre: Skipped 3 previous similar messages [ 5160.594698] LustreError: 165154:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5160.598140] LustreError: 165154:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5160.679806] Lustre: server umount lustre-MDT0000 complete [ 5161.953303] LustreError: 143505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5161.964261] LustreError: 143505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 100 previous similar messages [ 5163.733941] LustreError: 143484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344227 with bad export cookie 16511962666445840733 [ 5163.734502] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5163.754563] LustreError: 143484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5164.060404] Lustre: server umount lustre-MDT0001 complete [ 5177.732205] LustreError: 165556:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5177.764597] LustreError: 165556:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5177.852147] Lustre: server umount lustre-OST0000 complete [ 5191.781916] LustreError: 165758:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5191.785273] LustreError: 165758:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5191.953286] Lustre: server umount lustre-OST0001 complete [ 5208.692251] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5222.280593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5223.481232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5228.584676] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5239.489833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5244.773868] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5248.170975] Lustre: 168044:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5256.937554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5262.993874] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5268.747760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:845 to 0x280000401:865) [ 5272.860794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5274.963986] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:161) [ 5274.971301] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:841 to 0x2c0000401:865) [ 5278.982755] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5287.118468] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5290.531802] Lustre: 169893:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5342.589126] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 20:53:25 (1763344405) [ 5389.484876] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 20:54:12 (1763344452) [ 6889.087702] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 21:19:12 (1763345952) [ 6899.170260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6899.190952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.223658] Lustre: Skipped 3 previous similar messages [ 6899.227628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6899.369911] LustreError: 174929:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6899.374875] LustreError: 174929:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6899.574616] Lustre: server umount lustre-MDT0000 complete [ 6901.219262] LustreError: 168410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6901.233436] LustreError: 168410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6902.734043] LustreError: 166913:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345966 with bad export cookie 16511962666445849707 [ 6902.746243] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6902.760398] LustreError: 166913:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6902.862858] LustreError: 175130:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6902.866166] LustreError: 175130:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6903.119402] Lustre: server umount lustre-MDT0001 complete [ 6917.635232] LustreError: 175331:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6917.638578] LustreError: 175331:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6917.750806] Lustre: server umount lustre-OST0000 complete [ 6930.929415] LustreError: 175533:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6930.936616] LustreError: 175533:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6931.128944] Lustre: server umount lustre-OST0001 complete [ 6946.462925] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6955.701730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6956.224924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6956.228301] Lustre: Skipped 3 previous similar messages [ 6959.901777] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6968.351757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6972.733787] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6975.662400] Lustre: 177820:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6983.359163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6983.873374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6983.878939] Lustre: Skipped 1 previous similar message [ 6990.588337] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6993.064166] LustreError: 178176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6993.099374] LustreError: 178176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6999.834677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7001.610809] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:193) [ 7002.688687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5868 to 0x280000401:5889) [ 7002.705055] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5866 to 0x2c0000401:5889) [ 7006.559901] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7013.918901] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7017.532914] Lustre: 179671:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7048.752251] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 21:21:51 (1763346111) [ 7079.371434] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 21:22:22 (1763346142) [ 7109.247934] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 21:22:52 (1763346172) [ 7146.346851] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 21:23:29 (1763346209) [ 7181.256401] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 21:24:04 (1763346244) [ 7254.546064] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 21:25:16 (1763346316) [ 7289.400984] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 21:25:52 (1763346352) [ 7303.711861] LustreError: 178181:0:(qsd_handler.c:337:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x1 qsd:lustre-OST0001 qtype:usr id:60000 enforced:1 granted: 0 pending:0 waiting:1032 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 truncated:0 [ 7303.731797] LustreError: 178181:0:(qsd_handler.c:794:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:lustre-OST0001 qtype:usr id:60000 enforced:1 granted: 0 pending:0 waiting:0 req:0 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 truncated:0 [ 7353.504320] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 21:26:56 (1763346416) [ 7359.458698] Lustre: *** cfs_fail_loc=a09, val=0*** [ 7359.460818] Lustre: Skipped 1 previous similar message [ 7361.480223] Lustre: *** cfs_fail_loc=a09, val=0*** [ 7361.482610] Lustre: Skipped 105 previous similar messages [ 7365.508969] Lustre: *** cfs_fail_loc=a09, val=0*** [ 7365.513734] Lustre: Skipped 197 previous similar messages [ 7373.540445] Lustre: *** cfs_fail_loc=a09, val=0*** [ 7373.542516] Lustre: Skipped 361 previous similar messages [ 7389.550662] Lustre: *** cfs_fail_loc=a09, val=0*** [ 7389.554959] Lustre: Skipped 723 previous similar messages [ 7421.561078] Lustre: *** cfs_fail_loc=a09, val=0*** [ 7421.563111] Lustre: Skipped 1551 previous similar messages [ 7485.561572] Lustre: *** cfs_fail_loc=a09, val=0*** [ 7485.563605] Lustre: Skipped 3000 previous similar messages [ 7802.981748] Lustre: *** cfs_fail_loc=a08, val=0*** [ 7802.991617] Lustre: *** cfs_fail_loc=a08, val=0*** [ 7802.996154] Lustre: Skipped 2068 previous similar messages [ 7803.001777] Lustre: Skipped 7 previous similar messages [ 7832.674225] Lustre: *** cfs_fail_loc=a08, val=0*** [ 7832.676221] Lustre: Skipped 7 previous similar messages [ 7862.755773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7862.760253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7862.764020] Lustre: Skipped 4 previous similar messages [ 7867.873745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7867.882557] Lustre: Skipped 6 previous similar messages [ 7868.236934] LustreError: 191999:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7868.250253] LustreError: 191999:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7868.525523] Lustre: server umount lustre-MDT0000 complete [ 7872.088973] LustreError: 176688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346936 with bad export cookie 16511962666447624760 [ 7872.092770] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7872.111896] LustreError: 176688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7872.993390] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7872.997137] LustreError: 179437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7872.998023] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7873.006753] Lustre: Skipped 3 previous similar messages [ 7873.043227] LustreError: 179437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7878.113720] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7878.116367] Lustre: Skipped 2 previous similar messages [ 7878.353467] LustreError: 192202:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7878.357367] LustreError: 192202:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7878.592177] Lustre: server umount lustre-MDT0001 complete [ 7882.890661] LustreError: 192404:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7882.896591] LustreError: 192404:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7882.988992] Lustre: server umount lustre-OST0000 complete [ 7886.873805] Lustre: server umount lustre-OST0001 complete [ 7892.612846] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 7898.504181] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7909.482767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7916.529211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7921.854628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7927.467076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7938.213822] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7946.054405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7946.124418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7946.383610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7946.408532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7946.471965] Lustre: lustre-MDT0000: new disk, initializing [ 7946.552750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7946.556016] Lustre: Skipped 1 previous similar message [ 7946.569418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7950.232934] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7959.615721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7959.661990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7959.739279] Lustre: 195486:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7959.768094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7959.774270] Lustre: Skipped 1 previous similar message [ 7959.899601] Lustre: lustre-MDT0001: new disk, initializing [ 7959.991637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7960.048063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7960.056455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7963.379750] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7967.802521] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7974.461048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7974.538954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7974.791789] Lustre: lustre-OST0000: new disk, initializing [ 7974.796100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7974.882387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7976.671248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7976.675731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7976.789565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7980.449081] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7991.973423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7992.091274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7992.258661] Lustre: lustre-OST0001: new disk, initializing [ 7992.262178] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7992.346314] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7994.271051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7994.284051] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7994.378938] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7998.101756] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8007.383321] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8011.023064] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8040.638588] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 21:38:23 (1763347103) [ 8047.032397] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 21:38:30 (1763347110) [ 8074.448475] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 21:38:57 (1763347137) [ 8117.909873] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 21:39:40 (1763347180) [ 8137.583722] Lustre: DEBUG MARKER: rename directory return 255 [ 8174.921810] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 21:40:37 (1763347237) [ 8196.477817] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 21:40:59 (1763347259) [ 8232.067466] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 21:41:35 (1763347295) [ 8347.082767] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 21:43:29 (1763347409) [ 8373.829139] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 21:43:57 (1763347437) [ 8404.349649] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 21:44:27 (1763347467) [ 8527.773173] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 21:46:31 (1763347591) [ 8531.937770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8531.941486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8531.963064] Lustre: Skipped 3 previous similar messages [ 8531.990497] Lustre: Skipped 3 previous similar messages [ 8537.059621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8537.067211] Lustre: Skipped 3 previous similar messages [ 8538.241889] LustreError: 211676:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8538.247961] LustreError: 211676:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8538.418300] Lustre: server umount lustre-MDT0000 complete [ 8542.191134] LustreError: 195494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8542.250539] LustreError: 195494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8543.349116] LustreError: 195478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347607 with bad export cookie 16511962666447996985 [ 8543.357877] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8543.373502] LustreError: 195478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8543.689947] LustreError: 211881:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8543.724609] LustreError: 211881:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8543.993676] Lustre: server umount lustre-MDT0001 complete [ 8559.226131] LustreError: 212082:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8559.238402] LustreError: 212082:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8559.467184] Lustre: server umount lustre-OST0000 complete [ 8563.472104] Lustre: 3635:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763347611/real 1763347611] req@ffff9eff8808f480 x1848995146714368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763347627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8563.519387] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8563.922436] LustreError: 212285:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8563.930204] LustreError: 212285:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8564.118971] Lustre: server umount lustre-OST0001 complete [ 8586.675960] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 8596.744166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8597.164797] LustreError: 213659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8597.269918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8597.288877] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already [ 8601.460340] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8602.592705] LustreError: 213659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8610.141508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8610.531427] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already [ 8614.844554] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8617.917449] Lustre: 214771:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8624.047042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8624.446235] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already [ 8629.766455] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8635.687270] LustreError: 215126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8635.722894] LustreError: 215126:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8639.043342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8639.290512] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already [ 8642.672770] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65) [ 8642.682748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65) [ 8645.089949] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8653.778122] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8657.530431] Lustre: 216625:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8671.364602] LustreError: 213655:0:(osd_handler.c:3444:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95 [ 8677.345394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8677.357969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8677.375382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8680.779675] LustreError: 217017:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8680.784905] LustreError: 217017:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8680.877767] Lustre: server umount lustre-MDT0000 complete [ 8684.006697] LustreError: 213653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8684.016340] LustreError: 213653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8684.580378] LustreError: 213638:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347748 with bad export cookie 16511962666448045901 [ 8684.587510] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8684.626402] LustreError: 213638:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8684.934518] Lustre: server umount lustre-MDT0001 complete [ 8699.410203] LustreError: 217422:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 8699.440147] LustreError: 217422:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 8699.513941] Lustre: server umount lustre-OST0000 complete [ 8713.747545] Lustre: server umount lustre-OST0001 complete [ 8735.465810] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 8747.755561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.251855] LustreError: 219001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8748.416705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8748.425893] Lustre: Skipped 3 previous similar messages [ 8753.559794] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8762.551616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8767.455579] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8770.560182] Lustre: 220114:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8778.367566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8785.606138] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8794.088125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:97) [ 8794.419431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8795.722728] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8800.442652] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8808.699872] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8812.328636] Lustre: 221968:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8841.969061] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 21:51:44 (1763347904) [ 8887.664389] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61 [ 8889.463234] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 21:52:32 (1763347952) [ 8910.151598] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63 [ 8912.423824] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 21:52:55 (1763347975) [ 8951.364794] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65 [ 8952.987281] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 21:53:36 (1763348016) [ 8983.256294] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 21:54:06 (1763348046) [ 8996.550912] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 9002.123638] Lustre: 228195:0:(qsd_reint.c:241:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1) [ 9006.578501] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 9011.064712] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 9014.292687] Lustre: DEBUG MARKER: Write... [ 9032.844463] LustreError: 229411:0:(mgs_handler.c:1064:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1 [ 9039.077903] Lustre: DEBUG MARKER: Write... [ 9048.196402] Lustre: DEBUG MARKER: Write... [ 9106.304875] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 21:56:09 (1763348169) [ 9130.516562] LustreError: 233039:0:(mgs_handler.c:1064:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1 [ 9169.861212] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 21:57:13 (1763348233) [ 9195.837986] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 9197.041755] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 9274.755462] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 21:58:57 (1763348337) [ 9316.901561] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 21:59:40 (1763348380) [ 9351.771615] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 22:00:14 (1763348414) [ 9362.961194] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB) [ 9383.927544] Lustre: DEBUG MARKER: Write... [ 9385.672022] Lustre: DEBUG MARKER: Write out of block quota ... [ 9464.850530] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 22:02:08 (1763348528) [ 9477.379559] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB) [ 9553.460944] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 22:03:36 (1763348616) [ 9562.985453] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB) [ 9579.032798] Lustre: DEBUG MARKER: Write... [ 9581.489515] Lustre: DEBUG MARKER: Write out of block quota ... [ 9582.000656] LustreError: 218997:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 9622.151896] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 22:04:45 (1763348685) [ 9647.212485] Lustre: DEBUG MARKER: set to use default quota [ 9648.635337] Lustre: DEBUG MARKER: set default quota [ 9649.860622] Lustre: DEBUG MARKER: get default quota [ 9653.499592] Lustre: DEBUG MARKER: Test not out of quota [ 9656.293627] Lustre: DEBUG MARKER: Test out of quota [ 9663.155353] Lustre: DEBUG MARKER: Increase default quota [ 9677.498189] Lustre: DEBUG MARKER: Set quota to override default quota [ 9677.577638] LustreError: 218996:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1763953541 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 9684.266877] Lustre: DEBUG MARKER: Set to use default quota again [ 9700.575439] Lustre: DEBUG MARKER: Cleanup [ 9746.377922] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 22:06:49 (1763348809) [ 9766.356740] Lustre: DEBUG MARKER: set default quota for qpool1 [ 9767.914681] Lustre: DEBUG MARKER: Write from user that hasn't lqe [ 9806.448164] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 22:07:49 (1763348869) [ 9876.189366] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 22:08:59 (1763348939) [ 9973.514529] Lustre: DEBUG MARKER: Write... [ 9976.686327] Lustre: DEBUG MARKER: Write out of block quota ... [10084.995317] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 22:12:27 (1763349147) [10112.046617] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 22:12:55 (1763349175) [10118.382117] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 22:13:01 (1763349181) [10151.693827] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 22:13:35 (1763349215) [10186.130930] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 22:14:09 (1763349249) [10206.491954] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 22:14:29 (1763349269) [10228.585953] Lustre: *** cfs_fail_loc=a06, val=0*** [10228.588040] Lustre: Skipped 4 previous similar messages [10228.841402] LustreError: 219014:0:(qmt_lock.c:446:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11 [10228.841402] qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:23552 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [10234.334175] Lustre: Failing over lustre-OST0001 [10234.339822] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -19 [10234.344427] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10234.352247] LustreError: Skipped 1 previous similar message [10234.363175] Lustre: Skipped 3 previous similar messages [10234.365530] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10234.365541] Lustre: Skipped 4 previous similar messages [10234.428192] LustreError: 266879:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [10234.431442] LustreError: 266879:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [10234.519564] Lustre: server umount lustre-OST0001 complete [10235.376223] LustreError: 220468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10235.420328] LustreError: 220468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10242.538215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10242.860159] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10242.898888] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect [10244.092772] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect [10244.703092] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [10244.703248] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10244.708400] Lustre: Skipped 7 previous similar messages [10247.532157] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10286.036070] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 22:15:49 (1763349349) [10295.776159] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [10302.778418] LustreError: 269682:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms [10307.044847] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [10307.053915] Lustre: Skipped 1 previous similar message [10309.089803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10309.111346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10309.124627] Lustre: Skipped 1 previous similar message [10309.127428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10312.150198] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [10312.156031] Lustre: Skipped 3 previous similar messages [10312.808211] LustreError: 269682:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 awake [10312.928584] LustreError: 269832:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [10312.933598] LustreError: 269832:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10313.052590] Lustre: server umount lustre-MDT0000 complete [10314.721263] LustreError: 218995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10314.753968] LustreError: 218995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10322.716608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10322.926646] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10323.269393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10323.277649] Lustre: Skipped 3 previous similar messages [10323.358399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:112 to 0x280000401:129) [10323.366262] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:129) [10327.263622] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10328.547663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10328.558323] Lustre: Skipped 1 previous similar message [10351.777805] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 22:16:55 (1763349415) [10375.934811] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 22:17:18 (1763349438) [10396.843895] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 22:17:40 (1763349460) [10437.921212] Lustre: *** cfs_fail_loc=a08, val=0*** [10437.939644] Lustre: Skipped 8 previous similar messages [10437.944235] Lustre: *** cfs_fail_loc=a08, val=0*** [10511.956534] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 22:19:35 (1763349575) [10588.369865] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 22:20:51 (1763349651) [10647.902833] LustreError: 218994:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1763954512 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [10782.744582] LustreError: 223133:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1763954647 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [10926.326021] LustreError: 223133:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:16384 time:1763954790 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [11025.621597] Lustre: DEBUG MARKER: == sanity-quota test 87: lfs quota -a should print default quota setting ========================================================== 22:28:09 (1763350089) [11030.521651] Lustre: *** cfs_fail_loc=a09, val=0*** [11030.526324] Lustre: Skipped 1 previous similar message [11083.244704] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [11086.306185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11086.310589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11086.317268] Lustre: Skipped 4 previous similar messages [11087.914922] LustreError: 280462:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11087.918214] LustreError: 280462:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11088.002777] Lustre: server umount lustre-MDT0000 complete [11088.344704] LustreError: 218996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11088.382426] LustreError: 218996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [11091.428983] LustreError: 220479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11091.437306] LustreError: 220479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11095.435123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11095.567942] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11095.918928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11095.969092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [11095.978040] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [11099.494128] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11101.164417] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [11101.181952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11101.185688] Lustre: Skipped 2 previous similar messages [11112.790349] Lustre: DEBUG MARKER: == sanity-quota test 88: Writing over quota should not hang ========================================================== 22:29:36 (1763350176) [11114.057797] Lustre: DEBUG MARKER: SKIP: sanity-quota test_88 require client with >4k pages [11115.616812] Lustre: DEBUG MARKER: == sanity-quota test 89: Show default quota with squash_uid ========================================================== 22:29:39 (1763350179) [11165.665425] Lustre: DEBUG MARKER: == sanity-quota test 90a: lfs quota should work without mount point ========================================================== 22:30:28 (1763350228) [11191.619958] Lustre: DEBUG MARKER: == sanity-quota test 90b: lfs quota should work with multiple mount points ========================================================== 22:30:54 (1763350254) [11221.701197] Lustre: DEBUG MARKER: == sanity-quota test 91: new quota index files in quota_master ========================================================== 22:31:25 (1763350285) [11239.394262] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11239.400677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11239.400863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11239.425503] Lustre: Skipped 5 previous similar messages [11239.457254] Lustre: Skipped 6 previous similar messages [11242.191920] LustreError: 284921:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11242.203804] LustreError: 284921:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11242.392378] Lustre: server umount lustre-MDT0000 complete [11244.514150] LustreError: 220479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11244.537617] LustreError: 220479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11246.099825] LustreError: 229002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350310 with bad export cookie 16511962666449150333 [11246.103760] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11246.123905] LustreError: 229002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11249.634398] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11249.639054] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11249.654197] Lustre: Skipped 1 previous similar message [11249.669974] Lustre: Skipped 1 previous similar message [11252.404383] LustreError: 285123:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [11252.409408] LustreError: 285123:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11252.590734] Lustre: server umount lustre-MDT0001 complete [11255.825797] Lustre: server umount lustre-OST0000 complete [11258.866726] LustreError: 285525:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11258.870054] LustreError: 285525:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [11258.963275] Lustre: server umount lustre-OST0001 complete [11263.523759] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [11269.749423] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11278.444928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11285.040180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11290.023863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11295.741792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11301.148575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11301.214333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11301.439249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11301.468771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11301.534428] Lustre: lustre-MDT0000: new disk, initializing [11301.602518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11301.628178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11304.987484] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11311.442949] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11317.061530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11317.110752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11317.312126] Lustre: lustre-OST0000: new disk, initializing [11317.320091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11317.323496] Lustre: Skipped 1 previous similar message [11317.374391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11318.914707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11318.927162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11318.970539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11322.633301] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11328.793503] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11333.560366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11333.619831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11333.708478] Lustre: lustre-OST0001: new disk, initializing [11333.722604] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11333.779732] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11335.127555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [11335.140808] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [11335.175950] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [11338.817681] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11345.267507] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11356.136683] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [11356.139829] Lustre: Skipped 2 previous similar messages [11360.737207] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11361.899399] LustreError: 290491:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [11361.909513] LustreError: 290491:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11362.167719] Lustre: server umount lustre-MDT0000 complete [11373.424978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11373.504687] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11373.741434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11377.348379] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11381.223658] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11381.246927] Lustre: Skipped 3 previous similar messages [11384.137072] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11389.047027] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [11390.680176] LustreError: 291105:0:(qmt_entry.c:1166:qmt_map_lge_idx()) qmt: cannot map ostidx 0, num_used 1: rc = -22 [11392.797428] LustreError: 291104:0:(qmt_entry.c:1166:qmt_map_lge_idx()) qmt: cannot map ostidx 0, num_used 1: rc = -22 [11395.908993] LustreError: 291721:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [11395.913383] LustreError: 291721:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11396.093248] Lustre: server umount lustre-MDT0000 complete [11402.111300] LustreError: 287699:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350466 with bad export cookie 16511962666449153385 [11402.127760] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11412.320180] Lustre: 3638:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763350460/real 1763350460] req@ffff9f00b9ca1500 x1848995149616384/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763350476 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11412.344881] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11412.384755] Lustre: Skipped 1 previous similar message [11413.508861] Lustre: server umount lustre-OST0000 complete [11417.259733] Lustre: server umount lustre-OST0001 complete [11431.801479] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [11436.888577] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11446.768909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11454.013164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11459.360254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11465.031740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11475.683887] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11483.217035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11483.271351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11483.497206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11483.523906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11483.588914] Lustre: lustre-MDT0000: new disk, initializing [11483.666879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11483.685157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11486.531935] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11495.821384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11495.888996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11495.971840] Lustre: 295770:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11496.005226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11496.008811] Lustre: Skipped 1 previous similar message [11496.073364] Lustre: lustre-MDT0001: new disk, initializing [11496.174690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11496.183851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11499.660148] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11504.105356] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11509.847593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11509.931605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11510.133749] Lustre: lustre-OST0000: new disk, initializing [11510.140794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11512.184349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11512.195256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11512.253799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11515.374063] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11524.518571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11524.569898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11526.189465] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11526.246985] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11529.307550] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11538.353547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11542.163595] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11554.761597] Lustre: DEBUG MARKER: == sanity-quota test 92: Cannot set inode limit with Quota Pools ========================================================== 22:36:57 (1763350617) [11572.228596] Lustre: DEBUG MARKER: == sanity-quota test 93: update projid while client write to OST ========================================================== 22:37:15 (1763350635) [11593.070932] Lustre: *** cfs_fail_loc=170c, val=0*** [11637.625039] Lustre: DEBUG MARKER: == sanity-quota test 94: lfs quota all respects nodemap offset ========================================================== 22:38:20 (1763350700) [11701.217056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11701.232419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11701.254071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11701.257083] Lustre: Skipped 3 previous similar messages [11707.434926] LustreError: 302870:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [11707.439299] LustreError: 302870:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [11707.533700] Lustre: server umount lustre-MDT0000 complete [11710.433480] LustreError: 296653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11710.446443] LustreError: 296653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [11715.040858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11715.153166] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11715.371929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11715.374740] Lustre: Skipped 3 previous similar messages [11715.430316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [11719.096666] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11720.681428] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [11720.706974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11720.722641] Lustre: Skipped 2 previous similar messages [11727.852961] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [11727.867147] Lustre: Skipped 6 previous similar messages [11730.913092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11730.917868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11730.927325] Lustre: Skipped 5 previous similar messages [11732.397193] Lustre: server umount lustre-MDT0000 complete [11732.948415] LustreError: 295777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11732.970451] LustreError: 295777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11741.041283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.166729] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11741.625529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [11745.866836] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11746.789456] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [11752.784898] Lustre: DEBUG MARKER: == sanity-quota test 95a: Correct limits for squashed root ========================================================== 22:40:16 (1763350816) [11812.506174] Lustre: DEBUG MARKER: == sanity-quota test 95b: df respects squashed project id ========================================================== 22:41:14 (1763350874) [11871.895892] Lustre: DEBUG MARKER: == sanity-quota test 96: quota grant should be released when big files are deleted ========================================================== 22:42:15 (1763350935) [11873.615670] Lustre: DEBUG MARKER: SKIP: sanity-quota test_96 OST is too small, skip the test [11878.112836] Lustre: DEBUG MARKER: == sanity-quota test complete, duration 11627 sec ======== 22:42:21 (1763350941) [11879.875165] Lustre: DEBUG MARKER: === sanity-quota: start cleanup 22:42:22 (1763350942) === [11882.968098] Lustre: DEBUG MARKER: === sanity-quota: finish cleanup 22:42:26 (1763350946) === [11890.149313] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11890.154686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11890.161369] Lustre: Skipped 3 previous similar messages [11890.161576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11890.172683] Lustre: Skipped 7 previous similar messages [11892.961169] LustreError: 307883:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11892.977079] LustreError: 307883:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [11893.092990] Lustre: server umount lustre-MDT0000 complete [11895.266085] LustreError: 295777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.290299] LustreError: 295777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [11900.660843] LustreError: 295762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350964 with bad export cookie 16511962666449157221 [11900.668857] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11900.674155] LustreError: 295762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11900.959743] Lustre: server umount lustre-MDT0001 complete [11919.614672] Lustre: server umount lustre-OST0000 complete [11920.871686] Lustre: 3635:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763350969/real 1763350969] req@ffff9eff85576a00 x1848995149869568/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763350985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11926.496157] Lustre: 3636:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763350974/real 1763350974] req@ffff9eff88965c00 x1848995149869952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763350990 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11927.925810] Lustre: server umount lustre-OST0001 complete [11942.941322] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11945.573549] Key type lgssc unregistered [11945.837398] LNet: 310067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11945.841605] LNetError: 310067:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11945.863181] LNet: Removed LNI 192.168.201.108@tcp [11946.626242] Key type .llcrypt unregistered [11946.630936] Key type ._llcrypt unregistered