[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485467791 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002303] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007692] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010145] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014097] *** VALIDATE selinux *** [ 0.023332] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028527] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031109] *** VALIDATE tmpfs *** [ 0.033518] *** VALIDATE proc *** [ 0.035232] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.037288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041008] Speculative Store Bypass: Vulnerable [ 0.044130] debug: unmapping init [mem 0xffffffff94059000-0xffffffff94060fff] [ 0.046159] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047733] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048024] ... version: 2 [ 0.049014] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.055291] rcu: Hierarchical SRCU implementation. [ 0.057533] smp: Bringing up secondary CPUs ... [ 0.058585] x86: Booting SMP configuration: [ 0.059023] .... node #0, CPUs: #1 #2 #3 [ 0.063119] smp: Brought up 1 node, 4 CPUs [ 0.064973] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144528] node 0 deferred pages initialised in 77ms [ 0.147147] devtmpfs: initialized [ 0.148000] x86/mm: Memory block size: 128MB [ 0.150239] gcov: version magic: 0x41383552 [ 0.153408] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155267] pinctrl core: initialized pinctrl subsystem [ 0.156177] [ 0.156581] ************************************************************* [ 0.157013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158017] ** ** [ 0.159014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160013] ** ** [ 0.161012] ** This means that this kernel is built to expose internal ** [ 0.162013] ** IOMMU data structures, which may compromise security on ** [ 0.163014] ** your system. ** [ 0.164012] ** ** [ 0.165026] ** If you see this message and you are not debugging the ** [ 0.166015] ** kernel, report this immediately to your vendor! ** [ 0.167016] ** ** [ 0.168012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169014] ************************************************************* [ 0.170591] NET: Registered protocol family 16 [ 0.171420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174471] cpuidle: using governor menu [ 0.175000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178446] PCI: Using configuration type 1 for base access [ 0.180133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199125] cryptd: max_cpu_qlen set to 1000 [ 0.202273] ACPI: Added _OSI(Module Device) [ 0.204016] ACPI: Added _OSI(Processor Device) [ 0.206016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212014] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218590] ACPI: Interpreter enabled [ 0.219061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.220012] ACPI: Using IOAPIC for interrupt routing [ 0.221123] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222399] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231112] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.235022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.238092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.243124] acpiphp: Slot [2] registered [ 0.244140] acpiphp: Slot [3] registered [ 0.246093] acpiphp: Slot [4] registered [ 0.247113] acpiphp: Slot [5] registered [ 0.248155] acpiphp: Slot [6] registered [ 0.250108] acpiphp: Slot [7] registered [ 0.251086] acpiphp: Slot [8] registered [ 0.253100] acpiphp: Slot [9] registered [ 0.254099] acpiphp: Slot [10] registered [ 0.255141] acpiphp: Slot [11] registered [ 0.257102] acpiphp: Slot [12] registered [ 0.259123] acpiphp: Slot [13] registered [ 0.261118] acpiphp: Slot [14] registered [ 0.262097] acpiphp: Slot [15] registered [ 0.264110] acpiphp: Slot [16] registered [ 0.265083] acpiphp: Slot [17] registered [ 0.266117] acpiphp: Slot [18] registered [ 0.268104] acpiphp: Slot [19] registered [ 0.269102] acpiphp: Slot [20] registered [ 0.270108] acpiphp: Slot [21] registered [ 0.271103] acpiphp: Slot [22] registered [ 0.272092] acpiphp: Slot [23] registered [ 0.273103] acpiphp: Slot [24] registered [ 0.274111] acpiphp: Slot [25] registered [ 0.276077] acpiphp: Slot [26] registered [ 0.277082] acpiphp: Slot [27] registered [ 0.279137] acpiphp: Slot [28] registered [ 0.280085] acpiphp: Slot [29] registered [ 0.281092] acpiphp: Slot [30] registered [ 0.283090] acpiphp: Slot [31] registered [ 0.284104] PCI host bridge to bus 0000:00 [ 0.285016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.287018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.292020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.294021] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.296025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.298187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.302000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.306337] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.314013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.319010] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.321017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.323015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.326019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.328441] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.330733] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.333040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.335601] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.341013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.353014] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.358014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.364456] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.370021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.375014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.390015] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.399310] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.408017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.413014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.430015] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.439718] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.447016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.453017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.469016] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.479276] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.486031] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.494032] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.512026] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.522654] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.529017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.540020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.554017] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.562904] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.576025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.582026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.605034] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.616161] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.619386] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.621378] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.624360] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.626246] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.630126] iommu: Default domain type: Passthrough [ 0.632456] SCSI subsystem initialized [ 0.634208] ACPI: bus type USB registered [ 0.635106] usbcore: registered new interface driver usbfs [ 0.638118] usbcore: registered new interface driver hub [ 0.640080] usbcore: registered new device driver usb [ 0.642185] pps_core: LinuxPPS API ver. 1 registered [ 0.643012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.647069] PTP clock support registered [ 0.649081] EDAC MC: Ver: 3.0.0 [ 0.651151] PCI: Using ACPI for IRQ routing [ 0.652893] NetLabel: Initializing [ 0.653012] NetLabel: domain hash size = 128 [ 0.654010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.655106] NetLabel: unlabeled traffic allowed by default [ 0.657069] vgaarb: loaded [ 0.658282] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.660015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.667379] clocksource: Switched to clocksource kvm-clock [ 0.769671] VFS: Disk quotas dquot_6.6.0 [ 0.770836] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.773444] *** VALIDATE ramfs *** [ 0.774786] *** VALIDATE hugetlbfs *** [ 0.775943] pnp: PnP ACPI init [ 0.778066] pnp: PnP ACPI: found 6 devices [ 0.795399] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.798602] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.800341] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.802070] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.804177] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.806582] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.808824] NET: Registered protocol family 2 [ 0.811062] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.815452] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.818895] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.824492] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.828088] TCP: Hash tables configured (established 65536 bind 65536) [ 0.830974] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.834303] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.836917] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.839978] NET: Registered protocol family 1 [ 0.842737] RPC: Registered named UNIX socket transport module. [ 0.845182] RPC: Registered udp transport module. [ 0.847043] RPC: Registered tcp transport module. [ 0.848647] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.851215] NET: Registered protocol family 44 [ 0.852812] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.854669] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.856877] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.859296] PCI: CLS 0 bytes, default 64 [ 0.860608] Unpacking initramfs... [ 2.263577] debug: unmapping init [mem 0xffff97cefcc54000-0xffff97cefffbffff] [ 2.267654] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.269761] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.272568] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.771767] Initialise system trusted keyrings [ 2.773269] Key type blacklist registered [ 2.775490] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.784354] zbud: loaded [ 2.787205] *** VALIDATE nfs *** [ 2.788383] *** VALIDATE nfs4 *** [ 2.789530] pstore: using deflate compression [ 2.792215] Platform Keyring initialized [ 2.892227] NET: Registered protocol family 38 [ 2.893961] Key type asymmetric registered [ 2.895715] Asymmetric key parser 'x509' registered [ 2.897597] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.903840] io scheduler mq-deadline registered [ 2.905282] io scheduler kyber registered [ 2.906939] io scheduler bfq registered [ 2.908867] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.911718] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.914801] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.917716] ACPI: Power Button [PWRF] [ 3.010721] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.109734] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.314504] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.415890] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.606857] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.639263] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.668214] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.680477] Non-volatile memory driver v1.3 [ 3.682307] Linux agpgart interface v0.103 [ 3.720354] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.724271] vda: detected capacity change from 0 to 68526080 [ 3.739571] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.742848] vdb: detected capacity change from 0 to 1073741824 [ 3.760870] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.767788] vdc: detected capacity change from 0 to 2621440000 [ 3.781947] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.784929] vdd: detected capacity change from 0 to 2621440000 [ 3.799915] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.803083] vde: detected capacity change from 0 to 4294967296 [ 3.818152] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.820853] vdf: detected capacity change from 0 to 4294967296 [ 3.828101] libphy: Fixed MDIO Bus: probed [ 3.838883] usbcore: registered new interface driver usbserial_generic [ 3.840893] usbserial: USB Serial support registered for generic [ 3.842478] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.846765] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.848628] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.851036] mousedev: PS/2 mouse device common for all mice [ 3.854672] rtc_cmos 00:05: RTC can wake from S4 [ 3.857223] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.858105] rtc_cmos 00:05: registered as rtc0 [ 3.863119] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.864552] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.870270] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.870650] intel_pstate: CPU model not supported [ 3.877229] hid: raw HID events driver (C) Jiri Kosina [ 3.879494] usbcore: registered new interface driver usbhid [ 3.881078] usbhid: USB HID core driver [ 3.882541] drop_monitor: Initializing network drop monitor service [ 3.885152] Initializing XFRM netlink socket [ 3.887115] NET: Registered protocol family 10 [ 3.890654] Segment Routing with IPv6 [ 3.892481] NET: Registered protocol family 17 [ 3.894816] mpls_gso: MPLS GSO support [ 3.900787] RAS: Correctable Errors collector initialized. [ 3.903082] AVX version of gcm_enc/dec engaged. [ 3.905164] AES CTR mode by8 optimization enabled [ 3.998100] sched_clock: Marking stable (3998079992, 0)->(4932489859, -934409867) [ 4.001983] registered taskstats version 1 [ 4.004347] Loading compiled-in X.509 certificates [ 4.006533] zswap: loaded using pool lzo/zbud [ 4.030710] Key type big_key registered [ 4.044085] Key type encrypted registered [ 4.045756] ima: No TPM chip found, activating TPM-bypass! [ 4.047784] ima: Allocated hash algorithm: sha1 [ 4.049562] ima: No architecture policies found [ 4.051454] evm: Initialising EVM extended attributes: [ 4.053451] evm: security.selinux [ 4.054790] evm: security.ima [ 4.055809] evm: security.capability [ 4.057054] evm: HMAC attrs: 0x1 [ 4.059523] rtc_cmos 00:05: setting system clock to 2025-11-17 03:35:06 UTC (1763350506) [ 4.065506] debug: unmapping init [mem 0xffffffff95003000-0xffffffff951fffff] [ 4.068662] debug: unmapping init [mem 0xffffffff93d82000-0xffffffff94058fff] [ 4.077089] Write protecting the kernel read-only data: 28672k [ 4.080989] debug: unmapping init [mem 0xffffffff92403000-0xffffffff925fffff] [ 4.083729] debug: unmapping init [mem 0xffffffff92d14000-0xffffffff92dfffff] [ 4.134758] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.143450] systemd[1]: Detected virtualization kvm. [ 4.145105] systemd[1]: Detected architecture x86-64. [ 4.147111] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.174808] systemd[1]: No hostname configured. [ 4.176130] systemd[1]: Set hostname to . [ 4.178054] random: systemd: uninitialized urandom read (16 bytes read) [ 4.179819] systemd[1]: Initializing machine ID from random generator. [ 4.223185] random: ln: uninitialized urandom read (6 bytes read) [ 4.303292] random: systemd: uninitialized urandom read (16 bytes read) [ 4.306419] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.311530] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.319689] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.951548] device-mapper: uevent: version 1.0.3 [ 4.953969] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.633043] virtio_net virtio0 ens2: renamed from eth0 [ 5.646658] random: fast init done [ 5.723799] scsi host0: ata_piix [ 5.751180] scsi host1: ata_piix [ 5.752815] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.755292] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.552313] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.562086] random: crng init done [ 10.563097] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.046884] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.218697] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.504933] SELinux: Disabled at runtime. [ 12.572097] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.580561] systemd[1]: Detected virtualization kvm. [ 12.582475] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.156372] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.159901] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.164936] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.169415] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.172828] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.187821] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.200208] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK [0[ 13.255643] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [[ 13.678769] squashfs: version 4.0 (2009/01/31) Phillip Lougher  OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.016511] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.047538] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.249289] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.269696] EDAC sbridge: Ver: 1.1.2 [ 17.691276] Key type dns_resolver registered [ 18.537142] NFS: Registering the id_resolver key type [ 18.542210] Key type id_resolver registered [ 18.544657] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (5s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg417-server login: [ 34.736868] hrtimer: interrupt took 5851930 ns [ 77.634305] libcfs: loading out-of-tree module taints kernel. [ 77.662064] Key type ._llcrypt registered [ 77.663480] Key type .llcrypt registered [ 77.767967] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 96.126866] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 97.660745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 97.669988] alg: No test for adler32 (adler32-zlib) [ 99.191813] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 100.171137] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 101.988412] Key type lgssc registered [ 103.703368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 122.144841] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 123.100048] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 129.871240] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.841538] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 143.991446] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.593865] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 170.534292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.596251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 170.616695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.926733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.959552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.020696] Lustre: lustre-MDT0000: new disk, initializing [ 172.117658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.141170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 175.345827] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.510153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.597289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 187.707360] Lustre: 6510:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 187.743822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 187.748343] Lustre: Skipped 1 previous similar message [ 187.826583] Lustre: lustre-MDT0001: new disk, initializing [ 187.917361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.953268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.962764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 191.599293] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.853945] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 205.682319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.769206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 206.031699] Lustre: lustre-OST0000: new disk, initializing [ 206.045621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 206.123175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.526080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 211.534913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.541733] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.575151] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 224.368900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 224.449497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 224.555465] Lustre: lustre-OST0001: new disk, initializing [ 224.560499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 224.615208] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.050031] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.420210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 230.434961] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 230.478848] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 240.822091] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 250.779776] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 260.443023] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing check_logdir /tmp/testlogs/ [ 264.285851] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing yml_node [ 268.206666] Lustre: DEBUG MARKER: Client: 2.16.61.41 [ 270.417523] Lustre: DEBUG MARKER: MDS: 2.16.61.41 [ 272.880948] Lustre: DEBUG MARKER: OSS: 2.16.61.41 [ 274.417393] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-quota ============----- Sun Nov 16 22:39:35 EST 2025 [ 292.180579] Lustre: DEBUG MARKER: excepting tests: 2 4a 63 65 [ 294.103504] Lustre: DEBUG MARKER: skipping tests SLOW=no: 61 [ 297.409915] Lustre: DEBUG MARKER: === sanity-quota: start setup 22:39:58 (1763350798) === [ 302.389340] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing check_config_client /mnt/lustre [ 324.164097] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 328.067536] Lustre: 13288:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 331.608858] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 337.692708] Lustre: DEBUG MARKER: === sanity-quota: finish setup 22:40:38 (1763350838) === [ 382.519759] Lustre: DEBUG MARKER: == sanity-quota test 0: Test basic quota performance ===== 22:41:23 (1763350883) [ 423.237270] Lustre: DEBUG MARKER: == sanity-quota test 1a: Block hard limit (normal use and out of quota) ========================================================== 22:42:03 (1763350923) [ 432.714372] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 437.789795] Lustre: DEBUG MARKER: Write... [ 439.917085] Lustre: DEBUG MARKER: Write out of block quota ... [ 466.148808] Lustre: DEBUG MARKER: -------------------------------------- [ 467.672346] Lustre: DEBUG MARKER: Group quota (block hardlimit:10 MB) [ 473.826831] Lustre: DEBUG MARKER: Write... [ 476.171152] Lustre: DEBUG MARKER: Write out of block quota ... [ 506.179920] Lustre: DEBUG MARKER: -------------------------------------- [ 507.670892] Lustre: DEBUG MARKER: Project quota (block hardlimit:10 mb) [ 510.429724] Lustre: DEBUG MARKER: Write... [ 512.452642] Lustre: DEBUG MARKER: Write out of block quota ... [ 552.983790] Lustre: DEBUG MARKER: == sanity-quota test 1b: Quota pools: Block hard limit (normal use and out of quota) ========================================================== 22:44:13 (1763351053) [ 563.368527] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 581.384859] Lustre: DEBUG MARKER: Write... [ 583.516908] Lustre: DEBUG MARKER: Write out of block quota ... [ 610.591166] Lustre: DEBUG MARKER: -------------------------------------- [ 611.770789] Lustre: DEBUG MARKER: Group quota (block hardlimit:20 MB) [ 616.681788] Lustre: DEBUG MARKER: Write... [ 618.951715] Lustre: DEBUG MARKER: Write out of block quota ... [ 646.856405] Lustre: DEBUG MARKER: -------------------------------------- [ 648.522281] Lustre: DEBUG MARKER: Project quota (block hardlimit:20 mb) [ 651.107971] Lustre: DEBUG MARKER: Write... [ 653.177208] Lustre: DEBUG MARKER: Write out of block quota ... [ 699.508922] Lustre: DEBUG MARKER: == sanity-quota test 1c: Quota pools: check 3 pools with hardlimit only for global ========================================================== 22:46:40 (1763351200) [ 709.683392] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 733.673594] Lustre: DEBUG MARKER: Write... [ 736.301949] Lustre: DEBUG MARKER: Write out of block quota ... [ 807.094128] Lustre: DEBUG MARKER: == sanity-quota test 1d: Quota pools: check block hardlimit on different pools ========================================================== 22:48:27 (1763351307) [ 818.867242] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 841.429278] Lustre: DEBUG MARKER: Write... [ 843.654470] Lustre: DEBUG MARKER: Write out of block quota ... [ 910.661345] Lustre: DEBUG MARKER: == sanity-quota test 1e: Quota pools: global pool high block limit vs quota pool with small ========================================================== 22:50:11 (1763351411) [ 920.513989] Lustre: DEBUG MARKER: User quota (block hardlimit:53000000 MB) [ 931.640446] Lustre: DEBUG MARKER: Write... [ 933.419107] Lustre: DEBUG MARKER: Write out of block quota ... [ 942.899296] Lustre: DEBUG MARKER: Write... [ 979.420399] Lustre: DEBUG MARKER: == sanity-quota test 1f: Quota pools: correct qunit after removing/adding OST ========================================================== 22:51:20 (1763351480) [ 988.848948] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 998.354490] Lustre: DEBUG MARKER: Write... [ 999.837047] Lustre: DEBUG MARKER: Write out of block quota ... [ 1030.194614] Lustre: DEBUG MARKER: Write... [ 1031.895270] Lustre: DEBUG MARKER: Write out of block quota ... [ 1068.113953] Lustre: DEBUG MARKER: == sanity-quota test 1g: Quota pools: Block hard limit with wide striping ========================================================== 22:52:49 (1763351569) [ 1076.644345] Lustre: DEBUG MARKER: User quota (block hardlimit:40 MB) [ 1087.036458] Lustre: 6518:0:(osd_handler.c:2183:osd_trans_start()) lustre-MDT0000: credits 6546 > trans_max 3200 [ 1087.041800] Lustre: 6518:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 4/16/0, destroy: 0/0/0 [ 1087.046920] Lustre: 6518:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 401/401/0, xattr_set: 602/5615/0 [ 1087.050375] Lustre: 6518:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 20/118/0, punch: 0/0/0, quota 8/328/0 [ 1087.058455] Lustre: 6518:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/68/0, delete: 0/0/0 [ 1087.062660] Lustre: 6518:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1087.067867] CPU: 3 PID: 6518 Comm: mdt00_001 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1087.079466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1087.085742] Call Trace: [ 1087.091298] ? dump_stack+0xbb/0x10e [ 1087.093386] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [ 1087.096131] ? top_trans_start+0x599/0xd80 [ptlrpc] [ 1087.100608] ? lod_ref_add+0x30/0x30 [lod] [ 1087.103292] ? lod_trans_start+0x109/0x4c0 [lod] [ 1087.106222] ? mdd_declare_attr_set+0x190/0x690 [mdd] [ 1087.108624] ? mdd_env_info+0x25/0xc0 [mdd] [ 1087.110521] ? mdd_trans_start+0x18/0x30 [mdd] [ 1087.112270] ? mdd_attr_set+0xa5a/0x1250 [mdd] [ 1087.119183] ? mdt_reint_setattr+0x1342/0x1f90 [mdt] [ 1087.121364] ? mdt_reint_setattr+0x1342/0x1f90 [mdt] [ 1087.126337] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 1087.128918] ? mdt_reint_internal+0x6a0/0xdc0 [mdt] [ 1087.130520] ? mdt_reint+0x163/0x190 [mdt] [ 1087.134036] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 1087.139497] ? tgt_request_handle+0x573/0x1e70 [ptlrpc] [ 1087.142055] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 1087.145052] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 1087.148220] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 1087.150225] ? ptlrpc_wait_event+0x980/0x980 [ptlrpc] [ 1087.152946] ? kthread+0x1d1/0x200 [ 1087.154461] ? set_kthread_struct+0x70/0x70 [ 1087.155888] ? ret_from_fork+0x1f/0x30 [ 1088.682936] Lustre: DEBUG MARKER: Write... [ 1098.540536] Lustre: DEBUG MARKER: Write out of block quota ... [ 1154.684080] Lustre: DEBUG MARKER: == sanity-quota test 1h: Block hard limit test using fallocate ========================================================== 22:54:15 (1763351655) [ 1165.809282] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 1170.462670] Lustre: DEBUG MARKER: Write 5MiB Using Fallocate [ 1176.379746] Lustre: DEBUG MARKER: Write 11MiB Using Fallocate [ 1239.134246] Lustre: DEBUG MARKER: == sanity-quota test 1i: Quota pools: different limit and usage relations ========================================================== 22:55:40 (1763351740) [ 1248.389884] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 1259.925337] Lustre: DEBUG MARKER: Write... [ 1261.713716] Lustre: DEBUG MARKER: Write out of block quota ... [ 1285.703238] Lustre: DEBUG MARKER: Write... [ 1287.664876] Lustre: DEBUG MARKER: Write out of block quota ... [ 1296.804515] LustreError: 6518:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:15360 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 1331.541760] Lustre: DEBUG MARKER: == sanity-quota test 1j: Enable project quota enforcement for root ========================================================== 22:57:12 (1763351832) [ 1345.014591] Lustre: DEBUG MARKER: -------------------------------------- [ 1346.228946] Lustre: DEBUG MARKER: Project quota (hardlimit: 40 mb, 2048 files) [ 1693.078897] Lustre: DEBUG MARKER: SKIP: sanity-quota test_2 skipping excluded test 2 [ 1694.514560] Lustre: DEBUG MARKER: == sanity-quota test 3a: Block soft limit (start timer, timer goes off, stop timer) ========================================================== 23:03:15 (1763352195) [ 1732.091688] Lustre: DEBUG MARKER: Write after timer goes off [ 1733.494401] Lustre: DEBUG MARKER: Write after cancel lru locks [ 1794.865082] Lustre: DEBUG MARKER: Write after timer goes off [ 1795.821660] Lustre: DEBUG MARKER: Write after cancel lru locks [ 1851.369368] Lustre: DEBUG MARKER: Write after timer goes off [ 1853.396231] Lustre: DEBUG MARKER: Write after cancel lru locks [ 1894.056330] Lustre: DEBUG MARKER: == sanity-quota test 3b: Quota pools: Block soft limit (start timer, expires, stop timer) ========================================================== 23:06:35 (1763352395) [ 1934.532511] Lustre: DEBUG MARKER: Write after timer goes off [ 1936.743566] Lustre: DEBUG MARKER: Write after cancel lru locks [ 1993.023802] Lustre: DEBUG MARKER: Write after timer goes off [ 1995.159849] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2049.613891] Lustre: DEBUG MARKER: Write after timer goes off [ 2051.734142] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2100.577255] Lustre: DEBUG MARKER: == sanity-quota test 3c: Quota pools: check block soft limit on different pools ========================================================== 23:10:02 (1763352602) [ 2146.745649] Lustre: DEBUG MARKER: Write after timer goes off [ 2147.810791] Lustre: DEBUG MARKER: Write after cancel lru locks [ 2196.876272] Lustre: DEBUG MARKER: SKIP: sanity-quota test_4a skipping excluded test 4a [ 2197.768459] Lustre: DEBUG MARKER: == sanity-quota test 4b: Grace time strings handling ===== 23:11:39 (1763352699) [ 2203.616128] Lustre: DEBUG MARKER: == sanity-quota test 5: Chown [ 2269.248877] Lustre: DEBUG MARKER: == sanity-quota test 6: Test dropping acquire request on master ========================================================== 23:12:50 (1763352770) [ 2293.728458] Lustre: *** cfs_fail_loc=513, val=601*** [ 2293.730591] Lustre: Skipped 1 previous similar message [ 2294.801564] Lustre: *** cfs_fail_loc=513, val=601*** [ 2294.803606] Lustre: Skipped 16 previous similar messages [ 2295.337334] LustreError: 6521:0:(service.c:2319:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1849007123481856 [ 2297.312877] Lustre: *** cfs_fail_loc=513, val=601*** [ 2297.317964] Lustre: Skipped 20 previous similar messages [ 2300.897057] Lustre: *** cfs_fail_loc=513, val=601*** [ 2300.899295] Lustre: Skipped 19 previous similar messages [ 2306.016924] Lustre: *** cfs_fail_loc=513, val=601*** [ 2306.020264] Lustre: Skipped 21 previous similar messages [ 2310.624198] Lustre: 8410:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763352797/real 1763352797] req@ffff97cf6fe00e00 x1849007123481856/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1763352813 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_002.0' uid:0 gid:0 projid:4294967295 [ 2310.624587] Lustre: 18421:0:(service.c:1607:ptlrpc_at_send_early_reply()) @@@ Could not add any time (4/4), not sending early reply req@ffff97cf7d917100 x1849007107996800/t0(0) o4->6ae24c98-0e16-4c32-a8e8-f1b4b4ce5f82@192.168.204.17@tcp:487/0 lens 488/448 e 1 to 0 dl 1763352817 ref 2 fl Interpret:/600/0 rc 0/0 job:'dd.60000' uid:60000 gid:60000 projid:0 [ 2310.636464] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.663152] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 2310.668761] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2310.735518] LustreError: 6521:0:(service.c:2319:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1849007123490432 [ 2314.209828] Lustre: *** cfs_fail_loc=513, val=601*** [ 2314.211974] Lustre: Skipped 40 previous similar messages [ 2326.992158] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763352813/real 1763352813] req@ffff97cf42b22680 x1849007123490432/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1763352829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_002.0' uid:0 gid:0 projid:4294967295 [ 2327.008895] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.018684] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 2327.028029] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2329.904972] LustreError: 6521:0:(service.c:2319:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1849007123500544 [ 2331.616565] Lustre: *** cfs_fail_loc=513, val=601*** [ 2331.618405] Lustre: Skipped 90 previous similar messages [ 2346.464344] Lustre: 8410:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763352832/real 1763352832] req@ffff97ce4ff4ce00 x1849007123500544/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1763352848 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_002.0' uid:0 gid:0 projid:4294967295 [ 2346.475423] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.485326] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting [ 2346.489131] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2372.995537] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 23:14:34 (1763352874) [ 2387.754505] Lustre: Failing over lustre-OST0000 [ 2387.799147] LustreError: 74291:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2387.830967] Lustre: server umount lustre-OST0000 complete [ 2387.937908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2387.939969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.942633] LustreError: Skipped 1 previous similar message [ 2387.949804] Lustre: Skipped 1 previous similar message [ 2395.432857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2395.590983] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 2395.617436] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2396.450454] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2397.441483] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2397.441712] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2398.694167] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2402.202195] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2404.451467] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2406.628785] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2408.932486] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2411.394478] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2413.775577] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2429.418298] Lustre: Failing over lustre-OST0000 [ 2429.439182] LustreError: 77842:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2429.447851] LustreError: 77842:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2429.481419] Lustre: server umount lustre-OST0000 complete [ 2431.458267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.466225] LustreError: 39078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.472991] LustreError: 39078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2432.296947] LustreError: 39078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.594586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2433.737123] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 2433.749840] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2434.805893] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2435.394773] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2435.396086] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2435.399774] Lustre: Skipped 1 previous similar message [ 2436.327716] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2439.557989] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2441.517092] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2443.660580] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2445.751733] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2448.005244] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2450.211430] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2468.651830] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) ========================================================== 23:16:10 (1763352970) [ 2484.842104] Lustre: *** cfs_fail_loc=a02, val=0*** [ 2487.832195] LustreError: 3653:0:(qsd_handler.c:337:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8 qsd:lustre-OST0000 qtype:usr id:60000 enforced:1 granted: 1024 pending:0 waiting:0 req:1 usage: 2048 qunit:0 qtune:0 edquot:0 default:no revoke:0 truncated:0 [ 2487.843309] Lustre: Failing over lustre-OST0000 [ 2487.871119] LustreError: 82815:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2487.873992] LustreError: 82815:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2487.898402] Lustre: server umount lustre-OST0000 complete [ 2488.623131] LustreError: 39080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.337934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.347288] Lustre: Skipped 2 previous similar messages [ 2492.270277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2492.433620] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 2492.446961] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2493.729811] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2493.780981] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2493.781153] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2493.789037] Lustre: Skipped 1 previous similar message [ 2495.315925] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2498.895398] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2501.380122] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2503.854817] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2506.004915] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2508.192726] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2510.361051] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2530.257996] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ========================================================== 23:17:11 (1763353031) [ 2541.914589] LustreError: 87632:0:(qsd_reint.c:475:qsd_reint_main()) cfs_fail_timeout id a03 sleeping for 10000ms [ 2541.920666] LustreError: 87632:0:(qsd_reint.c:475:qsd_reint_main()) Skipped 4 previous similar messages [ 2542.855158] Lustre: Failing over lustre-MDT0000 [ 2543.072848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2543.079816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.087365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.179350] LustreError: 87734:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2543.182825] LustreError: 87734:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2543.248963] Lustre: server umount lustre-MDT0000 complete [ 2543.587554] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.594039] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2544.712088] LustreError: 87634:0:(qsd_reint.c:475:qsd_reint_main()) cfs_fail_timeout interrupted [ 2544.716507] LustreError: 87634:0:(qsd_reint.c:475:qsd_reint_main()) Skipped 3 previous similar messages [ 2547.719739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2547.795613] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2547.954320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.994256] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2549.793489] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2550.046373] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2553.040073] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2553.320788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2553.326028] Lustre: Skipped 1 previous similar message [ 2553.329658] LustreError: 3650:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x2020000:0x0].0x0 (ffff97ce43c89f00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2553.366080] Lustre: 87637:0:(qsd_reint.c:241:qsd_reint_index()) lustre-OST0001: index version for fid [0x200000005:0x100c:0x0] is 0, but index isn't empty (1) [ 2553.378450] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2553.407372] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:115 to 0x2c0000401:161) [ 2553.407626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:144 to 0x280000401:161) [ 2662.145737] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2664.319083] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2666.573576] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2668.722790] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2670.732635] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2686.154950] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ========================================================== 23:19:47 (1763353187) [ 2697.120977] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 2699.100091] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 2722.414610] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================================== 23:20:23 (1763353223) [ 2733.786140] Lustre: Failing over lustre-MDT0001 [ 2733.874585] LustreError: 98511:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2733.877296] LustreError: 98511:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2733.915528] Lustre: server umount lustre-MDT0001 complete [ 2734.380333] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.388165] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2737.634426] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.636331] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2737.639096] Lustre: Skipped 5 previous similar messages [ 2740.594472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2740.813475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2740.845666] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 2742.008940] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2742.495985] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2745.614976] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2745.828410] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2745.830663] Lustre: Skipped 3 previous similar messages [ 2745.839432] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2745.860846] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 2747.652855] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 2749.630993] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2751.725320] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 2753.959525] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2755.902086] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 2795.397043] Lustre: Failing over lustre-MDT0001 [ 2795.601110] LustreError: 101706:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2795.604037] LustreError: 101706:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2795.635296] Lustre: server umount lustre-MDT0001 complete [ 2797.028496] LustreError: 9302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.028781] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2797.035546] LustreError: 9302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2799.154469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2799.379436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2799.405433] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 2800.910847] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2801.131282] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2803.805483] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2804.714582] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2804.736386] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 2805.821456] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 2807.913645] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2810.041256] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 2811.997416] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2813.888781] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 2851.710139] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ==== 23:22:33 (1763353353) [ 3025.127916] Lustre: DEBUG MARKER: == sanity-quota test 9: Block limit larger than 4GB (b10707) ========================================================== 23:25:26 (1763353526) [ 3025.787079] Lustre: DEBUG MARKER: OST0_SIZE: 3600244 required: 4900000 [ 3028.778179] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ======== 23:25:30 (1763353530) [ 3050.118752] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota ======= 23:25:51 (1763353551) [ 3069.997622] Lustre: DEBUG MARKER: == sanity-quota test 12a: Block quota rebalancing ======== 23:26:11 (1763353571) [ 3106.205165] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ======== 23:26:47 (1763353607) [ 3163.162162] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================================== 23:27:44 (1763353664) [ 3187.882568] Lustre: DEBUG MARKER: == sanity-quota test 14: check panic in qmt_site_recalc_cb ========================================================== 23:28:09 (1763353689) [ 3198.034331] Lustre: Failing over lustre-OST0000 [ 3198.060716] LustreError: 115912:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3198.062762] LustreError: 115912:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3198.095111] Lustre: server umount lustre-OST0000 complete [ 3198.946413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3198.947947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.949869] LustreError: Skipped 1 previous similar message [ 3198.954294] Lustre: Skipped 3 previous similar messages [ 3198.957921] LustreError: 105577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.963507] LustreError: 105577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3204.504203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3204.652533] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 3204.666498] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3205.407610] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3206.462415] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3206.462488] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3206.466389] Lustre: Skipped 5 previous similar messages [ 3207.259424] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3226.009526] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ========= 23:28:47 (1763353727) [ 3234.762888] Lustre: DEBUG MARKER: == sanity-quota test 16a: lfs quota should skip the inactive MDT/OST ========================================================== 23:28:56 (1763353736) [ 3242.274115] Lustre: lustre-MDT0001: Client 6ae24c98-0e16-4c32-a8e8-f1b4b4ce5f82 (at 192.168.204.17@tcp) reconnecting [ 3255.486145] Lustre: DEBUG MARKER: == sanity-quota test 16b: lfs quota should skip the nonexistent MDT/OST ========================================================== 23:29:17 (1763353757) [ 3256.025444] Lustre: DEBUG MARKER: SKIP: sanity-quota test_16b needs >= 3 MDTs [ 3256.620246] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error == 23:29:18 (1763353758) [ 3264.917818] Lustre: *** cfs_fail_loc=a04, val=37*** [ 3264.919641] LustreError: 8409:0:(qsd_handler.c:337:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x1 qsd:lustre-OST0001 qtype:usr id:60000 enforced:1 granted: 0 pending:0 waiting:1056 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 truncated:0 [ 3265.954000] Lustre: *** cfs_fail_loc=a04, val=37*** [ 3265.955438] LustreError: 8409:0:(qsd_handler.c:337:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x1 qsd:lustre-OST0001 qtype:usr id:60000 enforced:1 granted: 0 pending:0 waiting:1056 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0 truncated:0 [ 3291.429595] Lustre: *** cfs_fail_loc=a04, val=11*** [ 3314.353974] Lustre: *** cfs_fail_loc=a04, val=110*** [ 3314.355515] Lustre: Skipped 1 previous similar message [ 3338.142129] Lustre: *** cfs_fail_loc=a04, val=107*** [ 3338.143754] Lustre: Skipped 1 previous similar message [ 3372.046060] Lustre: DEBUG MARKER: == sanity-quota test 18: MDS failover while writing, no watchdog triggered (b14840) ========================================================== 23:31:13 (1763353873) [ 3377.559188] Lustre: DEBUG MARKER: User quota (limit: 200) [ 3379.111430] Lustre: DEBUG MARKER: Write 100M (buffered) ... [ 3381.797454] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3382.408259] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 3383.114651] Lustre: Failing over lustre-MDT0000 [ 3383.376382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3383.379402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.384495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3383.459933] LustreError: 129198:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3383.462282] LustreError: 129198:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3383.501411] Lustre: server umount lustre-MDT0000 complete [ 3383.776908] LustreError: 9302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3383.783093] LustreError: 9302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3398.127612] LDISKFS-fs (dm-0): 7 truncates cleaned up [ 3398.129253] LDISKFS-fs (dm-0): recovery complete [ 3398.133264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.169275] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3398.249237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3398.285305] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3399.663758] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3403.041152] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3403.749486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3403.751283] Lustre: Skipped 1 previous similar message [ 3403.759892] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3403.775641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1351 to 0x280000401:1377) [ 3403.775647] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1348 to 0x2c0000401:1377) [ 3405.054543] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3405.712130] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3414.319626] Lustre: DEBUG MARKER: (dd_pid=110456, time=6, timeout=600) [ 3426.475725] Lustre: DEBUG MARKER: User quota (limit: 200) [ 3427.978553] Lustre: DEBUG MARKER: Write 100M (directio) ... [ 3430.705504] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3431.256249] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 3431.926731] Lustre: Failing over lustre-MDT0000 [ 3432.280930] Lustre: server umount lustre-MDT0000 complete [ 3434.465084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3434.468209] LustreError: Skipped 1 previous similar message [ 3446.537153] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3446.538779] LDISKFS-fs (dm-0): recovery complete [ 3446.542427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3446.577691] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3446.656270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3447.995855] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3451.900763] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1348 to 0x2c0000401:1409) [ 3451.900769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 3453.168000] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3453.771337] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3456.261689] Lustre: DEBUG MARKER: (dd_pid=112766, time=0, timeout=600) [ 3469.204811] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 23:32:50 (1763353970) [ 3487.695629] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 23:33:09 (1763353989) [ 3495.531300] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [ 3499.502064] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [ 3500.103962] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [ 3500.709950] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000 [ 3501.432729] Lustre: DEBUG MARKER: Set quota for 1 times [ 3503.165329] Lustre: DEBUG MARKER: Set quota for 2 times [ 3504.970146] Lustre: DEBUG MARKER: Set quota for 3 times [ 3506.853555] Lustre: DEBUG MARKER: Set quota for 4 times [ 3508.668590] Lustre: DEBUG MARKER: Set quota for 5 times [ 3510.417624] Lustre: DEBUG MARKER: Set quota for 6 times [ 3512.241602] Lustre: DEBUG MARKER: Set quota for 7 times [ 3514.029871] Lustre: DEBUG MARKER: Set quota for 8 times [ 3515.946852] Lustre: DEBUG MARKER: Set quota for 9 times [ 3517.846249] Lustre: DEBUG MARKER: Set quota for 10 times [ 3519.544611] Lustre: DEBUG MARKER: Set quota for 11 times [ 3521.146481] Lustre: DEBUG MARKER: Set quota for 12 times [ 3522.768778] Lustre: DEBUG MARKER: Set quota for 13 times [ 3524.513927] Lustre: DEBUG MARKER: Set quota for 14 times [ 3526.173366] Lustre: DEBUG MARKER: Set quota for 15 times [ 3527.791330] Lustre: DEBUG MARKER: Set quota for 16 times [ 3529.424618] Lustre: DEBUG MARKER: Set quota for 17 times [ 3547.946819] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 23:34:09 (1763354049) [ 3556.866326] LustreError: 139710:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3556.868545] LustreError: 139710:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3556.894646] Lustre: server umount lustre-MDT0000 complete [ 3558.126804] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354060 with bad export cookie 9563701473882951385 [ 3558.128722] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3558.130497] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3558.235896] Lustre: server umount lustre-MDT0001 complete [ 3574.560157] Lustre: 3651:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763354061/real 1763354061] req@ffff97ce51285880 x1849007127071104/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763354077 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3578.656075] Lustre: 3651:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763354065/real 1763354065] req@ffff97ce51285180 x1849007127071744/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763354081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3578.666509] Lustre: 3651:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3578.943280] Lustre: server umount lustre-OST0000 complete [ 3580.225478] Lustre: server umount lustre-OST0001 complete [ 3585.786704] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 3589.593191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.736233] LustreError: 141484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3589.743671] LustreError: 141484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 3589.767423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.015329] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3593.853735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3595.244324] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3596.234826] Lustre: 142604:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3598.495440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3599.653278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 3600.684973] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3603.799416] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3605.871911] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97) [ 3605.875657] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1412 to 0x2c0000401:1441) [ 3605.918707] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3609.082614] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3610.271734] Lustre: 144457:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3624.417506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3624.419808] Lustre: Skipped 4 previous similar messages [ 3629.536642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3629.538266] Lustre: Skipped 3 previous similar messages [ 3629.716360] Lustre: server umount lustre-MDT0000 complete [ 3630.942741] LustreError: 142216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354133 with bad export cookie 9563701473882960261 [ 3630.944350] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3630.946927] LustreError: 142216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3631.049223] Lustre: server umount lustre-MDT0001 complete [ 3641.865432] Lustre: server umount lustre-OST0000 complete [ 3653.340129] Lustre: server umount lustre-OST0001 complete [ 3658.448961] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 3662.147421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3662.324863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3662.326824] Lustre: Skipped 3 previous similar messages [ 3663.629480] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3666.636465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3668.114685] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3669.107364] Lustre: 148237:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3671.379796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3672.547234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1473) [ 3673.519541] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3676.466649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3677.539469] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1412 to 0x2c0000401:1473) [ 3677.541144] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:129) [ 3678.451863] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3681.555789] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3682.760977] Lustre: 150084:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3691.226483] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 23:36:32 (1763354192) [ 3691.766947] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144 [ 3692.304995] Lustre: DEBUG MARKER: run for 4MB test file [ 3697.415479] Lustre: DEBUG MARKER: User quota (limit: 4 MB) [ 3699.041384] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT [ 3699.593172] Lustre: DEBUG MARKER: Write half of file [ 3700.219588] Lustre: DEBUG MARKER: Write out of block quota ... [ 3700.862792] Lustre: DEBUG MARKER: Step1: done [ 3701.470279] Lustre: DEBUG MARKER: Step2: rewrite should succeed [ 3702.083974] Lustre: DEBUG MARKER: Step2: done [ 3718.765114] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440 [ 3719.303678] Lustre: DEBUG MARKER: run for 40MB test file [ 3723.274311] Lustre: DEBUG MARKER: User quota (limit: 40 MB) [ 3724.796212] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT [ 3725.297094] Lustre: DEBUG MARKER: Write half of file [ 3726.207380] Lustre: DEBUG MARKER: Write out of block quota ... [ 3727.139697] Lustre: DEBUG MARKER: Step1: done [ 3727.667220] Lustre: DEBUG MARKER: Step2: rewrite should succeed [ 3728.211720] Lustre: DEBUG MARKER: Step2: done [ 3750.788023] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 23:37:32 (1763354252) [ 3768.723377] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 23:37:50 (1763354270) [ 3781.053451] Lustre: DEBUG MARKER: Write... [ 3781.779963] Lustre: DEBUG MARKER: Write out of block quota ... [ 3809.643297] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 23:38:31 (1763354311) [ 3811.909036] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 23:38:33 (1763354313) [ 3816.326454] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 23:38:38 (1763354318) [ 3820.031797] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 23:38:41 (1763354321) [ 3823.377303] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 23:38:45 (1763354325) [ 3850.885462] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user [ 3897.192894] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user [ 3963.057270] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 23:41:04 (1763354464) [ 3982.821235] Lustre: DEBUG MARKER: Restart... [ 3984.352541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3984.354478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.357200] Lustre: Skipped 13 previous similar messages [ 3984.359036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3989.986240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3989.988378] Lustre: Skipped 6 previous similar messages [ 3990.142659] LustreError: 170538:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3990.144967] LustreError: 170538:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 3990.175317] Lustre: server umount lustre-MDT0000 complete [ 3991.353846] LustreError: 148240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354493 with bad export cookie 9563701473882962564 [ 3991.356974] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3991.357668] LustreError: 148240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3991.473434] Lustre: server umount lustre-MDT0001 complete [ 4003.331714] Lustre: server umount lustre-OST0000 complete [ 4013.782473] Lustre: server umount lustre-OST0001 complete [ 4018.771101] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 4022.272097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.424462] LustreError: 172314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4022.431060] LustreError: 172314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4022.455476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.458174] Lustre: Skipped 3 previous similar messages [ 4023.816185] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4026.665045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4028.112838] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4029.145096] Lustre: 173426:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4031.455281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4032.610629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1484 to 0x280000401:1505) [ 4033.545062] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4036.381872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4037.476376] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:161) [ 4037.477223] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1482 to 0x2c0000401:1505) [ 4038.305576] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4041.286757] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4042.414265] Lustre: 175272:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4072.711036] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 23:42:54 (1763354574) [ 4097.490640] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 23:43:19 (1763354599) [ 4553.634347] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 23:50:55 (1763355055) [ 4559.328953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4559.332955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4559.337020] Lustre: Skipped 3 previous similar messages [ 4559.338675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4563.020802] LustreError: 180522:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4563.022926] LustreError: 180522:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4563.105495] Lustre: server umount lustre-MDT0000 complete [ 4564.374096] LustreError: 179320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355066 with bad export cookie 9563701473882971279 [ 4564.377961] LustreError: 179320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4564.378059] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4564.495643] Lustre: server umount lustre-MDT0001 complete [ 4576.016982] Lustre: server umount lustre-OST0000 complete [ 4587.565921] Lustre: server umount lustre-OST0001 complete [ 4593.137416] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 4597.196594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.370087] LustreError: 182296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4597.378681] LustreError: 182296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4597.402759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4597.405129] Lustre: Skipped 3 previous similar messages [ 4598.858106] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4601.871664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4603.308352] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4604.270223] Lustre: 183407:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4606.565602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4607.715268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6508 to 0x280000401:6529) [ 4608.680783] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4611.668543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4612.773124] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6506 to 0x2c0000401:6529) [ 4612.775701] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:193) [ 4613.791613] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4616.997195] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4637.986764] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 23:52:19 (1763355139) [ 4653.066407] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 23:52:34 (1763355154) [ 4668.165496] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 23:52:49 (1763355169) [ 4684.089866] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 23:53:05 (1763355185) [ 4699.505424] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 23:53:21 (1763355201) [ 4735.016623] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 23:53:56 (1763355236) [ 4746.319601] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 23:54:08 (1763355248) [ 4776.183508] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 23:54:37 (1763355277) [ 4778.223403] Lustre: *** cfs_fail_loc=a09, val=0*** [ 4778.225222] Lustre: Skipped 1 previous similar message [ 4779.228772] Lustre: *** cfs_fail_loc=a09, val=0*** [ 4779.230014] Lustre: Skipped 167 previous similar messages [ 4781.240567] Lustre: *** cfs_fail_loc=a09, val=0*** [ 4781.242593] Lustre: Skipped 343 previous similar messages [ 4785.246605] Lustre: *** cfs_fail_loc=a09, val=0*** [ 4785.247716] Lustre: Skipped 641 previous similar messages [ 4793.253771] Lustre: *** cfs_fail_loc=a09, val=0*** [ 4793.255303] Lustre: Skipped 1259 previous similar messages [ 4809.256428] Lustre: *** cfs_fail_loc=a09, val=0*** [ 4809.258881] Lustre: Skipped 2509 previous similar messages [ 4945.376624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4945.378923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4945.380706] Lustre: Skipped 3 previous similar messages [ 4950.285533] Lustre: server umount lustre-MDT0000 complete [ 4951.571499] LustreError: 185260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355454 with bad export cookie 9563701473884713838 [ 4951.575721] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4951.576730] LustreError: 185260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4957.759907] Lustre: server umount lustre-MDT0001 complete [ 4959.340849] Lustre: server umount lustre-OST0000 complete [ 4960.939582] Lustre: server umount lustre-OST0001 complete [ 4963.103067] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 4965.773358] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 4970.043892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4972.938814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4975.142366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4977.585733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4982.532130] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 4986.147744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4986.173487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4986.279433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4986.294632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4986.334401] Lustre: lustre-MDT0000: new disk, initializing [ 4986.361691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4986.363604] Lustre: Skipped 3 previous similar messages [ 4986.370670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4987.738645] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4992.015070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4992.039099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4992.067183] Lustre: 201559:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4992.072940] Lustre: 201559:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 4992.083328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4992.085239] Lustre: Skipped 1 previous similar message [ 4992.127390] Lustre: lustre-MDT0001: new disk, initializing [ 4992.159245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4992.162288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4993.598025] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4995.940770] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4998.282397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4998.307652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4998.395083] Lustre: lustre-OST0000: new disk, initializing [ 4998.397751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5000.435454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5000.439695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5000.471776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5000.506677] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5004.880345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5004.910192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5004.950231] Lustre: lustre-OST0001: new disk, initializing [ 5004.952115] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5006.255665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5006.258348] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5006.280336] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5007.011680] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5011.612517] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5012.817098] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5026.989732] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 23:58:48 (1763355528) [ 5029.210314] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 23:58:50 (1763355530) [ 5046.058288] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 23:59:07 (1763355547) [ 5067.110975] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 23:59:28 (1763355568) [ 5072.940183] Lustre: DEBUG MARKER: rename directory return 255 [ 5092.774819] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 23:59:54 (1763355594) [ 5099.661832] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 00:00:01 (1763355601) [ 5109.362430] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 00:00:11 (1763355611) [ 5138.473274] LustreError: 205995:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 5153.815052] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 00:00:55 (1763355655) [ 5162.581253] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 00:01:04 (1763355664) [ 5179.957170] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 00:01:21 (1763355681) [ 5225.081472] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 00:02:06 (1763355726) [ 5226.465920] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5226.466773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5226.469840] Lustre: Skipped 12 previous similar messages [ 5226.472622] Lustre: Skipped 7 previous similar messages [ 5232.319099] LustreError: 218576:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5232.320826] LustreError: 218576:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 5232.354970] Lustre: server umount lustre-MDT0000 complete [ 5233.536035] LustreError: 201550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355735 with bad export cookie 9563701473885085909 [ 5233.539523] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5233.660357] Lustre: server umount lustre-MDT0001 complete [ 5244.935527] Lustre: server umount lustre-OST0000 complete [ 5256.411784] Lustre: server umount lustre-OST0001 complete [ 5262.741127] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 5266.241234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5266.394300] LustreError: 220552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5266.399089] LustreError: 220552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5266.416597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5266.418465] Lustre: Skipped 3 previous similar messages [ 5266.420864] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already [ 5267.695841] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5270.584375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5270.688212] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already [ 5271.949607] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5272.921911] Lustre: 221664:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5275.073795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5275.175067] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already [ 5276.194749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65) [ 5277.117463] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5279.995967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5280.049899] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already [ 5281.062474] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65) [ 5282.124375] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5285.232098] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5293.553810] LustreError: 220548:0:(osd_handler.c:3444:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95 [ 5296.611629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5296.611993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5296.613371] Lustre: Skipped 8 previous similar messages [ 5300.884540] Lustre: server umount lustre-MDT0000 complete [ 5302.136812] LustreError: 220533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355804 with bad export cookie 9563701473885134601 [ 5302.139341] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5302.142103] LustreError: 220533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5302.252168] Lustre: server umount lustre-MDT0001 complete [ 5314.048375] Lustre: server umount lustre-OST0000 complete [ 5325.518415] Lustre: server umount lustre-OST0001 complete [ 5332.095533] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 5335.836829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5337.310563] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5340.141370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5341.714211] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5344.997561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5346.147290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5347.028278] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5349.875939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5350.948482] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:97) [ 5351.917738] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5355.041952] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5375.624225] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 00:04:37 (1763355877) [ 5396.442526] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61 [ 5396.973419] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 00:04:58 (1763355898) [ 5403.680768] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63 [ 5404.283805] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 00:05:05 (1763355905) [ 5420.685407] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65 [ 5421.178121] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 00:05:22 (1763355922) [ 5436.656948] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 00:05:38 (1763355938) [ 5440.151640] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 5441.599344] Lustre: 235454:0:(qsd_reint.c:241:qsd_reint_index()) lustre-OST0001: index version for fid [0x200000005:0x100b:0x0] is 0, but index isn't empty (1) [ 5441.603348] Lustre: 235454:0:(qsd_reint.c:241:qsd_reint_index()) Skipped 1 previous similar message [ 5442.912190] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [ 5444.535240] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 [ 5445.700701] Lustre: DEBUG MARKER: Write... [ 5453.774547] LustreError: 236664:0:(mgs_handler.c:1064:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1 [ 5457.449132] Lustre: DEBUG MARKER: Write... [ 5460.388626] Lustre: DEBUG MARKER: Write... [ 5497.781442] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 00:06:39 (1763355999) [ 5508.220882] LustreError: 240673:0:(mgs_handler.c:1064:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1 [ 5527.745696] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 00:07:09 (1763356029) [ 5538.228307] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB) [ 5538.767186] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 5565.214761] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 00:07:46 (1763356066) [ 5584.172083] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 00:08:05 (1763356085) [ 5599.294265] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 00:08:21 (1763356101) [ 5602.793965] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB) [ 5614.885253] Lustre: DEBUG MARKER: Write... [ 5615.521882] Lustre: DEBUG MARKER: Write out of block quota ... [ 5662.178455] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 00:09:23 (1763356163) [ 5665.753071] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB) [ 5703.447915] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 00:10:05 (1763356205) [ 5707.111573] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB) [ 5714.014184] Lustre: DEBUG MARKER: Write... [ 5714.614240] Lustre: DEBUG MARKER: Write out of block quota ... [ 5714.715256] LustreError: 225882:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 5743.580536] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 00:10:45 (1763356245) [ 5754.500391] Lustre: DEBUG MARKER: set to use default quota [ 5754.960689] Lustre: DEBUG MARKER: set default quota [ 5755.452770] Lustre: DEBUG MARKER: get default quota [ 5756.890877] Lustre: DEBUG MARKER: Test not out of quota [ 5757.956990] Lustre: DEBUG MARKER: Test out of quota [ 5760.711739] Lustre: DEBUG MARKER: Increase default quota [ 5769.020090] Lustre: DEBUG MARKER: Set quota to override default quota [ 5769.039393] LustreError: 225878:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1763961071 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 5771.845605] Lustre: DEBUG MARKER: Set to use default quota again [ 5780.552603] Lustre: DEBUG MARKER: Cleanup [ 5807.768106] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 00:11:49 (1763356309) [ 5817.251875] Lustre: DEBUG MARKER: set default quota for qpool1 [ 5817.740376] Lustre: DEBUG MARKER: Write from user that hasn't lqe [ 5841.676576] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 00:12:23 (1763356343) [ 5872.853283] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 00:12:54 (1763356374) [ 5937.062765] Lustre: DEBUG MARKER: Write... [ 5938.007105] Lustre: DEBUG MARKER: Write out of block quota ... [ 6014.856279] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 00:15:16 (1763356516) [ 6030.581866] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 00:15:32 (1763356532) [ 6032.945729] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 00:15:34 (1763356534) [ 6051.093220] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 00:15:52 (1763356552) [ 6072.346381] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 00:16:13 (1763356573) [ 6082.818553] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 00:16:24 (1763356584) [ 6093.564304] Lustre: *** cfs_fail_loc=a06, val=0*** [ 6093.565571] Lustre: Skipped 3075 previous similar messages [ 6093.634261] LustreError: 225897:0:(qmt_lock.c:446:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11 [ 6093.634261] qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:23552 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [ 6097.366564] Lustre: Failing over lustre-OST0001 [ 6097.398073] LustreError: 276624:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6097.400485] LustreError: 276624:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 6097.425728] Lustre: server umount lustre-OST0001 complete [ 6098.405340] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6098.408632] Lustre: Skipped 4 previous similar messages [ 6098.410443] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6098.410541] LustreError: 227358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6098.415864] LustreError: 227358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 6100.481675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6100.567795] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6100.576695] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect [ 6100.578499] Lustre: Skipped 1 previous similar message [ 6101.680910] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6101.682875] Lustre: Skipped 1 previous similar message [ 6102.494879] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6102.516730] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6102.516781] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6102.520439] Lustre: Skipped 7 previous similar messages [ 6102.522907] Lustre: Skipped 1 previous similar message [ 6129.826511] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 00:17:11 (1763356631) [ 6133.408927] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB) [ 6138.350036] LustreError: 279616:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms [ 6141.409352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6141.410877] Lustre: Skipped 3 previous similar messages [ 6148.392114] LustreError: 279616:0:(qmt_pool.c:1295:qmt_pool_recalc()) cfs_fail_timeout id a07 awake [ 6148.472314] Lustre: server umount lustre-MDT0000 complete [ 6151.267221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6151.311370] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6151.388287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.390136] Lustre: Skipped 7 previous similar messages [ 6151.429269] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:129) [ 6151.432397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:112 to 0x280000401:129) [ 6152.659950] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6156.771918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6156.774313] Lustre: Skipped 1 previous similar message [ 6166.440214] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 00:17:48 (1763356668) [ 6174.962967] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 00:17:56 (1763356676) [ 6182.336319] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 00:18:04 (1763356684) [ 6199.471762] Lustre: *** cfs_fail_loc=a08, val=0*** [ 6199.473176] Lustre: Skipped 8 previous similar messages [ 6199.475181] Lustre: *** cfs_fail_loc=a08, val=0*** [ 6199.476207] Lustre: Skipped 1 previous similar message [ 6237.782133] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 00:18:59 (1763356739) [ 6250.709612] LustreError: 225882:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:3072 soft:0 granted:3072 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 6277.684679] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 00:19:39 (1763356779) [ 6294.743607] LustreError: 226601:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1763961597 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 6337.337888] LustreError: 225879:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:16384 time:1763961639 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 6390.673074] LustreError: 225878:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:16385 time:1763961693 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no [ 6430.909089] Lustre: DEBUG MARKER: == sanity-quota test 87: lfs quota -a should print default quota setting ========================================================== 00:22:12 (1763356932) [ 6433.085268] Lustre: *** cfs_fail_loc=a09, val=0*** [ 6433.086474] Lustre: Skipped 1 previous similar message [ 6453.899984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6453.901638] Lustre: Skipped 7 previous similar messages [ 6460.101692] Lustre: server umount lustre-MDT0000 complete [ 6463.562912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.628949] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6463.835579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:132 to 0x280000401:161) [ 6463.836176] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:113 to 0x2c0000401:161) [ 6465.366931] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6469.094368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6469.096661] Lustre: Skipped 2 previous similar messages [ 6469.098099] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 6471.714350] Lustre: DEBUG MARKER: == sanity-quota test 88: Writing over quota should not hang ========================================================== 00:22:53 (1763356973) [ 6472.263710] Lustre: DEBUG MARKER: SKIP: sanity-quota test_88 require client with >4k pages [ 6472.833941] Lustre: DEBUG MARKER: == sanity-quota test 89: Show default quota with squash_uid ========================================================== 00:22:54 (1763356974) [ 6510.641896] Lustre: DEBUG MARKER: == sanity-quota test 90a: lfs quota should work without mount point ========================================================== 00:23:32 (1763357012) [ 6521.020750] Lustre: DEBUG MARKER: == sanity-quota test 90b: lfs quota should work with multiple mount points ========================================================== 00:23:42 (1763357022) [ 6532.000376] Lustre: DEBUG MARKER: == sanity-quota test 91: new quota index files in quota_master ========================================================== 00:23:53 (1763357033) [ 6545.892849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6545.896180] Lustre: Skipped 11 previous similar messages [ 6549.656174] Lustre: server umount lustre-MDT0000 complete [ 6550.900703] LustreError: 225865:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357053 with bad export cookie 9563701473886243226 [ 6550.903776] LustreError: 225865:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6550.904362] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6565.344174] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6565.440522] Lustre: server umount lustre-MDT0001 complete [ 6575.072893] LustreError: 295341:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff97ce502d6300 x1849007139113856/t0(0) o101->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 456/496 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0 projid:4294967295 [ 6575.078502] LustreError: 295341:0:(qsd_reint.c:38:qsd_reint_completion()) lustre-OST0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x20000:0x0], rc:-5 [ 6575.084294] LustreError: 295341:0:(qsd_reint.c:38:qsd_reint_completion()) Skipped 2 previous similar messages [ 6581.216127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6581.249791] Lustre: server umount lustre-OST0000 complete [ 6582.432494] LustreError: 295346:0:(qsd_reint.c:38:qsd_reint_completion()) lustre-OST0001: failed to enqueue global quota lock, glb fid:[0x200000006:0x2020000:0x0], rc:-5 [ 6582.435820] LustreError: 295346:0:(qsd_reint.c:38:qsd_reint_completion()) Skipped 2 previous similar messages [ 6588.558863] Lustre: server umount lustre-OST0001 complete [ 6590.519520] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 6592.933753] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 6596.687834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6599.226899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6601.337625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6603.513603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6605.582920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6605.606690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.688394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6605.698694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6605.736139] Lustre: lustre-MDT0000: new disk, initializing [ 6605.764778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6607.037712] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6610.091375] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6612.270365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6612.294013] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6612.379243] Lustre: lustre-OST0000: new disk, initializing [ 6612.381089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6612.383286] Lustre: Skipped 1 previous similar message [ 6614.192954] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6614.194538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6614.198508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6614.207517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6617.200936] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6619.524829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6619.556170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6619.590789] Lustre: lustre-OST0001: new disk, initializing [ 6619.592458] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6621.495784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 6621.498328] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 6621.506459] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 6621.552182] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6624.757983] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6631.905422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6631.908232] Lustre: Skipped 9 previous similar messages [ 6636.600424] Lustre: server umount lustre-MDT0000 complete [ 6640.318410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6640.354370] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6641.754180] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6643.524706] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6644.104846] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6651.745771] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6651.748222] Lustre: Skipped 3 previous similar messages [ 6660.166088] LustreError: 301128:0:(qmt_entry.c:1166:qmt_map_lge_idx()) qmt: cannot map ostidx 0, num_used 1: rc = -22 [ 6668.704640] Lustre: server umount lustre-MDT0000 complete [ 6670.806685] LustreError: 299560:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357173 with bad export cookie 9563701473886246110 [ 6670.811710] LustreError: 299560:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6670.845048] Lustre: server umount lustre-OST0000 complete [ 6672.104430] Lustre: server umount lustre-OST0001 complete [ 6678.152649] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 6680.407704] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 6684.461136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6687.051232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6689.098288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6691.421441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6695.720408] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 6699.049882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6699.077613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6699.179755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6699.193810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6699.233753] Lustre: lustre-MDT0000: new disk, initializing [ 6699.264380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6700.679776] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6705.318910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6705.350596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6705.387655] Lustre: 305803:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6705.391737] Lustre: 305803:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 3 previous similar messages [ 6705.400958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6705.402552] Lustre: Skipped 1 previous similar message [ 6705.445579] Lustre: lustre-MDT0001: new disk, initializing [ 6705.472355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6705.474683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6706.830212] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6709.239900] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6711.522503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6711.551323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6712.943557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6712.965668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6713.597192] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6717.922947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6717.946224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6717.984589] Lustre: lustre-OST0001: new disk, initializing [ 6717.986300] Lustre: Skipped 1 previous similar message [ 6717.988589] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6717.991386] Lustre: Skipped 1 previous similar message [ 6719.789628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6719.793054] Lustre: Skipped 1 previous similar message [ 6719.794951] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6719.804773] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6719.937994] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6724.287545] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6725.574527] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6733.101505] Lustre: DEBUG MARKER: == sanity-quota test 92: Cannot set inode limit with Quota Pools ========================================================== 00:27:14 (1763357234) [ 6741.412780] Lustre: DEBUG MARKER: == sanity-quota test 93: update projid while client write to OST ========================================================== 00:27:23 (1763357243) [ 6751.233431] Lustre: *** cfs_fail_loc=170c, val=0*** [ 6780.132796] Lustre: DEBUG MARKER: == sanity-quota test 94: lfs quota all respects nodemap offset ========================================================== 00:28:01 (1763357281) [ 6824.743567] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.17@tcp (stopping) [ 6824.746952] Lustre: Skipped 3 previous similar messages [ 6827.489575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6827.500162] Lustre: Skipped 20 previous similar messages [ 6829.516946] LustreError: 312885:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6829.522684] LustreError: 312885:0:(obd_class.h:479:obd_check_dev()) Skipped 63 previous similar messages [ 6829.554748] Lustre: server umount lustre-MDT0000 complete [ 6829.853720] LustreError: 305808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6829.861181] LustreError: 305808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6833.103126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6833.143253] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6833.147178] LustreError: Skipped 1 previous similar message [ 6833.224286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6833.226444] Lustre: Skipped 9 previous similar messages [ 6833.273192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6834.847893] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6838.251696] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 6843.363918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6843.367554] LustreError: Skipped 1 previous similar message [ 6845.015936] Lustre: server umount lustre-MDT0000 complete [ 6847.908567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.070504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 6849.442445] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6852.777371] Lustre: DEBUG MARKER: == sanity-quota test 95a: Correct limits for squashed root ========================================================== 00:29:14 (1763357354) [ 6853.097501] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 6891.687028] Lustre: DEBUG MARKER: == sanity-quota test 95b: df respects squashed project id ========================================================== 00:29:53 (1763357393) [ 6923.017682] Lustre: DEBUG MARKER: == sanity-quota test 96: quota grant should be released when big files are deleted ========================================================== 00:30:24 (1763357424) [ 6923.553720] Lustre: DEBUG MARKER: SKIP: sanity-quota test_96 OST is too small, skip the test [ 6925.670102] Lustre: DEBUG MARKER: == sanity-quota test complete, duration 6651 sec ========= 00:30:27 (1763357427) [ 6926.178794] Lustre: DEBUG MARKER: === sanity-quota: start cleanup 00:30:27 (1763357427) === [ 6927.263781] Lustre: DEBUG MARKER: === sanity-quota: finish cleanup 00:30:28 (1763357428) === [ 6934.439743] Lustre: server umount lustre-MDT0000 complete [ 6937.989144] LustreError: 305795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357440 with bad export cookie 9563701473886249911 [ 6937.992481] LustreError: 305795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6938.120625] Lustre: server umount lustre-MDT0001 complete [ 6952.189108] Lustre: server umount lustre-OST0000 complete [ 6965.013697] Lustre: server umount lustre-OST0001 complete [ 6970.021551] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 6970.932851] Key type lgssc unregistered [ 6971.048255] LNet: 320032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6971.050172] LNetError: 320032:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6971.057272] LNet: Removed LNI 192.168.204.117@tcp [ 6971.366092] Key type .llcrypt unregistered [ 6971.367109] Key type ._llcrypt unregistered