[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496076791 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002383] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007463] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013071] *** VALIDATE selinux *** [ 0.021112] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026884] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027222] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028150] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029129] *** VALIDATE tmpfs *** [ 0.030557] *** VALIDATE proc *** [ 0.031356] *** VALIDATE cgroup *** [ 0.032013] *** VALIDATE cgroup2 *** [ 0.033318] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036032] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.040208] debug: unmapping init [mem 0xffffffff95e59000-0xffffffff95e60fff] [ 0.043291] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044798] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045030] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047019] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049019] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051011] ... event mask: 000000070000000f [ 0.052270] rcu: Hierarchical SRCU implementation. [ 0.054408] smp: Bringing up secondary CPUs ... [ 0.055493] x86: Booting SMP configuration: [ 0.056019] .... node #0, CPUs: #1 #2 #3 [ 0.059074] smp: Brought up 1 node, 4 CPUs [ 0.061008] smpboot: Max logical packages: 1 [ 0.062009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146816] node 0 deferred pages initialised in 82ms [ 0.148125] devtmpfs: initialized [ 0.149284] x86/mm: Memory block size: 128MB [ 0.153052] gcov: version magic: 0x41383552 [ 0.156290] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158270] pinctrl core: initialized pinctrl subsystem [ 0.159165] [ 0.159633] ************************************************************* [ 0.160015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161011] ** ** [ 0.162015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163012] ** ** [ 0.164009] ** This means that this kernel is built to expose internal ** [ 0.165011] ** IOMMU data structures, which may compromise security on ** [ 0.166010] ** your system. ** [ 0.167013] ** ** [ 0.168010] ** If you see this message and you are not debugging the ** [ 0.169011] ** kernel, report this immediately to your vendor! ** [ 0.170010] ** ** [ 0.171010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172009] ************************************************************* [ 0.173635] NET: Registered protocol family 16 [ 0.174424] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177426] cpuidle: using governor menu [ 0.179066] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181515] PCI: Using configuration type 1 for base access [ 0.184131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.196025] cryptd: max_cpu_qlen set to 1000 [ 0.198237] ACPI: Added _OSI(Module Device) [ 0.199017] ACPI: Added _OSI(Processor Device) [ 0.200015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.201013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.205520] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209063] ACPI: Interpreter enabled [ 0.210051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211011] ACPI: Using IOAPIC for interrupt routing [ 0.212092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.213406] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222459] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.224017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.225084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.227363] acpiphp: Slot [2] registered [ 0.228098] acpiphp: Slot [5] registered [ 0.229109] acpiphp: Slot [6] registered [ 0.230107] acpiphp: Slot [7] registered [ 0.231099] acpiphp: Slot [8] registered [ 0.232077] acpiphp: Slot [9] registered [ 0.233127] acpiphp: Slot [10] registered [ 0.234106] acpiphp: Slot [3] registered [ 0.235098] acpiphp: Slot [4] registered [ 0.236083] acpiphp: Slot [11] registered [ 0.237078] acpiphp: Slot [12] registered [ 0.238079] acpiphp: Slot [13] registered [ 0.239104] acpiphp: Slot [14] registered [ 0.240078] acpiphp: Slot [15] registered [ 0.241098] acpiphp: Slot [16] registered [ 0.242079] acpiphp: Slot [17] registered [ 0.243084] acpiphp: Slot [18] registered [ 0.244100] acpiphp: Slot [19] registered [ 0.245116] acpiphp: Slot [20] registered [ 0.246136] acpiphp: Slot [21] registered [ 0.247129] acpiphp: Slot [22] registered [ 0.248079] acpiphp: Slot [23] registered [ 0.249081] acpiphp: Slot [24] registered [ 0.250091] acpiphp: Slot [25] registered [ 0.251087] acpiphp: Slot [26] registered [ 0.252119] acpiphp: Slot [27] registered [ 0.253140] acpiphp: Slot [28] registered [ 0.254099] acpiphp: Slot [29] registered [ 0.255082] acpiphp: Slot [30] registered [ 0.256085] acpiphp: Slot [31] registered [ 0.257051] PCI host bridge to bus 0000:00 [ 0.258015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.259017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.260017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.261017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.262017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.263022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.264158] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.266101] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.268466] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.274691] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.277653] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.278024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.279028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.280035] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.281548] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.282711] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.283041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.284469] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.291020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.307021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.313020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.320000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.328023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.335020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.360021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.375872] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.392016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.402016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.417064] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.427014] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.437017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.445014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.464017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.473804] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.482016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.494018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.518019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.532811] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.539013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.547019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.599015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.615327] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.629016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.639014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.655021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.665382] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.667354] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.670373] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.672235] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.674175] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.678163] iommu: Default domain type: Passthrough [ 0.680498] SCSI subsystem initialized [ 0.682147] ACPI: bus type USB registered [ 0.684131] usbcore: registered new interface driver usbfs [ 0.686088] usbcore: registered new interface driver hub [ 0.688101] usbcore: registered new device driver usb [ 0.690196] pps_core: LinuxPPS API ver. 1 registered [ 0.692012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.694072] PTP clock support registered [ 0.696280] EDAC MC: Ver: 3.0.0 [ 0.698139] PCI: Using ACPI for IRQ routing [ 0.700116] NetLabel: Initializing [ 0.702012] NetLabel: domain hash size = 128 [ 0.703009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.705094] NetLabel: unlabeled traffic allowed by default [ 0.707146] vgaarb: loaded [ 0.709270] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.711009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.718220] clocksource: Switched to clocksource kvm-clock [ 0.829978] VFS: Disk quotas dquot_6.6.0 [ 0.831145] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.832862] *** VALIDATE ramfs *** [ 0.833781] *** VALIDATE hugetlbfs *** [ 0.834911] pnp: PnP ACPI init [ 0.836681] pnp: PnP ACPI: found 6 devices [ 0.855181] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.857471] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.859629] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.862111] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.864794] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.867515] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.870927] NET: Registered protocol family 2 [ 0.873431] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.879995] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.882936] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.888223] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.891775] TCP: Hash tables configured (established 65536 bind 65536) [ 0.894665] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.897900] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.900369] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.903335] NET: Registered protocol family 1 [ 0.906298] RPC: Registered named UNIX socket transport module. [ 0.908518] RPC: Registered udp transport module. [ 0.910514] RPC: Registered tcp transport module. [ 0.912332] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.914506] NET: Registered protocol family 44 [ 0.916251] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.918920] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.921097] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.923640] PCI: CLS 0 bytes, default 64 [ 0.925276] Unpacking initramfs... [ 2.284355] debug: unmapping init [mem 0xffff97e27cc54000-0xffff97e27ffbffff] [ 2.288594] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.290842] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.294031] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.775462] Initialise system trusted keyrings [ 2.777596] Key type blacklist registered [ 2.779417] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.788717] zbud: loaded [ 2.791809] *** VALIDATE nfs *** [ 2.792764] *** VALIDATE nfs4 *** [ 2.794228] pstore: using deflate compression [ 2.797311] Platform Keyring initialized [ 2.902719] NET: Registered protocol family 38 [ 2.904569] Key type asymmetric registered [ 2.906261] Asymmetric key parser 'x509' registered [ 2.907996] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.911211] io scheduler mq-deadline registered [ 2.912859] io scheduler kyber registered [ 2.914581] io scheduler bfq registered [ 2.916766] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.920288] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.922871] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.925473] ACPI: Power Button [PWRF] [ 2.930400] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.937263] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.948084] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.955570] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.974483] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.001339] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.028090] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.032231] Non-volatile memory driver v1.3 [ 3.034166] Linux agpgart interface v0.103 [ 3.061221] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.064132] vda: detected capacity change from 0 to 74760192 [ 3.081805] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.085247] vdb: detected capacity change from 0 to 1073741824 [ 3.101502] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.104900] vdc: detected capacity change from 0 to 2621440000 [ 3.120966] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.124365] vdd: detected capacity change from 0 to 2621440000 [ 3.142411] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.144547] vde: detected capacity change from 0 to 4294967296 [ 3.160112] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.163370] vdf: detected capacity change from 0 to 4294967296 [ 3.171871] libphy: Fixed MDIO Bus: probed [ 3.178441] usbcore: registered new interface driver usbserial_generic [ 3.182692] usbserial: USB Serial support registered for generic [ 3.185228] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.189592] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.191439] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.194352] mousedev: PS/2 mouse device common for all mice [ 3.197738] rtc_cmos 00:05: RTC can wake from S4 [ 3.200795] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.205309] rtc_cmos 00:05: registered as rtc0 [ 3.207272] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.210417] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.211113] intel_pstate: CPU model not supported [ 3.218028] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.218357] hid: raw HID events driver (C) Jiri Kosina [ 3.224207] usbcore: registered new interface driver usbhid [ 3.226458] usbhid: USB HID core driver [ 3.228313] drop_monitor: Initializing network drop monitor service [ 3.231229] Initializing XFRM netlink socket [ 3.233425] NET: Registered protocol family 10 [ 3.236414] Segment Routing with IPv6 [ 3.237919] NET: Registered protocol family 17 [ 3.239944] mpls_gso: MPLS GSO support [ 3.246204] RAS: Correctable Errors collector initialized. [ 3.247864] AVX version of gcm_enc/dec engaged. [ 3.248981] AES CTR mode by8 optimization enabled [ 3.316147] sched_clock: Marking stable (3316116510, 0)->(4219136409, -903019899) [ 3.320229] registered taskstats version 1 [ 3.322145] Loading compiled-in X.509 certificates [ 3.323709] zswap: loaded using pool lzo/zbud [ 3.346284] Key type big_key registered [ 3.360624] Key type encrypted registered [ 3.362181] ima: No TPM chip found, activating TPM-bypass! [ 3.364394] ima: Allocated hash algorithm: sha1 [ 3.366300] ima: No architecture policies found [ 3.368271] evm: Initialising EVM extended attributes: [ 3.370261] evm: security.selinux [ 3.371765] evm: security.ima [ 3.373010] evm: security.capability [ 3.374344] evm: HMAC attrs: 0x1 [ 3.378891] rtc_cmos 00:05: setting system clock to 2026-08-21 18:53:10 UTC (1787338390) [ 3.384785] debug: unmapping init [mem 0xffffffff96e03000-0xffffffff96ffffff] [ 3.387601] debug: unmapping init [mem 0xffffffff95b82000-0xffffffff95e58fff] [ 3.398089] Write protecting the kernel read-only data: 28672k [ 3.401622] debug: unmapping init [mem 0xffffffff94203000-0xffffffff943fffff] [ 3.404279] debug: unmapping init [mem 0xffffffff94b14000-0xffffffff94bfffff] [ 3.436762] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.444328] systemd[1]: Detected virtualization kvm. [ 3.446702] systemd[1]: Detected architecture x86-64. [ 3.448794] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.471165] systemd[1]: No hostname configured. [ 3.472974] systemd[1]: Set hostname to . [ 3.475201] random: systemd: uninitialized urandom read (16 bytes read) [ 3.477883] systemd[1]: Initializing machine ID from random generator. [ 3.515718] random: ln: uninitialized urandom read (6 bytes read) [ 3.598991] random: systemd: uninitialized urandom read (16 bytes read) [ 3.602218] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.607410] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.616960] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.164578] device-mapper: uevent: version 1.0.3 [ 4.166136] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.770980] virtio_net virtio0 ens2: renamed from eth0 [ 4.813098] random: fast init done [ 4.900775] scsi host0: ata_piix [ 4.913306] scsi host1: ata_piix [ 4.914645] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.916993] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.881744] random: crng init done [ 8.882774] random: 7 urandom warning(s) missed due to ratelimiting [ 9.118256] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.643674] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.724299] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.992467] SELinux: Disabled at runtime. [ 11.055780] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.064101] systemd[1]: Detected virtualization kvm. [ 11.065887] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.522426] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.526257] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.534318] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.539138] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.542553] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.549895] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.553676] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 11.640265] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 11.898100] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.187276] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.331565] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.352589] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.366929] EDAC sbridge: Ver: 1.1.2 [ 13.713974] Key type dns_resolver registered [ 13.989868] NFS: Registering the id_resolver key type [ 13.991946] Key type id_resolver registered [ 13.993080] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg208-server login: [ 41.845087] hrtimer: interrupt took 5509216 ns [ 64.546046] libcfs: loading out-of-tree module taints kernel. [ 64.642065] Key type ._llcrypt registered [ 64.645887] Key type .llcrypt registered [ 64.882912] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 83.568292] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 84.967650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.977951] alg: No test for adler32 (adler32-zlib) [ 86.584831] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 87.486297] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 89.248259] Key type lgssc registered [ 91.230582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.571431] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 154.179931] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 167.383188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 167.399805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 168.709638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 168.745642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 168.814112] Lustre: lustre-MDT0000: new disk, initializing [ 168.881970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.903916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.980900] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.302478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.405689] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 187.440692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 187.445797] Lustre: Skipped 1 previous similar message [ 187.582702] Lustre: lustre-MDT0001: new disk, initializing [ 187.658214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.689181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.704159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 192.136272] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.217683] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 208.845580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.084608] Lustre: lustre-OST0000: new disk, initializing [ 209.087681] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.092709] Lustre: 8454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.153476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.122363] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.617671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 216.633562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 216.727024] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 231.564697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.745876] Lustre: lustre-OST0001: new disk, initializing [ 231.755522] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 231.763714] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.849555] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 238.607359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 238.617963] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 238.715718] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 238.981621] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.068406] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 260.389886] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 267.014974] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing check_logdir /tmp/testlogs/ [ 272.845928] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing yml_node [ 278.915855] Lustre: DEBUG MARKER: Client: 2.17.57.44 [ 282.148994] Lustre: DEBUG MARKER: MDS: 2.17.57.44 [ 285.377216] Lustre: DEBUG MARKER: OSS: 2.17.57.44 [ 287.160866] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-scrub ============----- Fri Aug 21 14:57:52 EDT 2026 [ 307.668140] Lustre: DEBUG MARKER: excepting tests: [ 320.281697] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 330.720723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.739828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.851397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.862495] Lustre: Skipped 3 previous similar messages [ 336.130717] Lustre: server umount lustre-MDT0000 complete [ 344.669614] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787338731 with bad export cookie 6487046033638116763 [ 344.675959] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 344.687710] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 345.219912] Lustre: server umount lustre-MDT0001 complete [ 362.400326] Lustre: 3651:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787338733/real 1787338733] req@ffff97e2c440d500 x1874160227821952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787338749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 362.417588] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.429341] Lustre: Skipped 3 previous similar messages [ 364.619989] Lustre: server umount lustre-OST0000 complete [ 365.344101] Lustre: 3649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787338736/real 1787338736] req@ffff97e2f794ad80 x1874160227822208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787338752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 367.585412] Lustre: 3651:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787338738/real 1787338738] req@ffff97e2ff834700 x1874160227822464/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787338754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 370.529160] Lustre: 3649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787338741/real 1787338741] req@ffff97e2ff83ce00 x1874160227822848/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787338757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 374.479550] Lustre: server umount lustre-OST0001 complete [ 390.560205] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 393.912675] Key type lgssc unregistered [ 394.285338] LNet: 14803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.296472] LNetError: 14803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 394.321264] LNet: Removed LNI 192.168.202.108@tcp [ 395.485456] Key type .llcrypt unregistered [ 395.488393] Key type ._llcrypt unregistered [ 423.284277] Key type ._llcrypt registered [ 423.285509] Key type .llcrypt registered [ 423.376342] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 435.802618] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 436.674526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 436.750638] alg: No test for adler32 (adler32-zlib) [ 437.988614] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 438.350050] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 440.160160] Key type lgssc registered [ 441.359940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 491.651972] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 506.569293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 506.587887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.045865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 508.107137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 508.232300] Lustre: lustre-MDT0000: new disk, initializing [ 508.426355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.485279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 514.379517] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 528.980373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.157657] Lustre: 19250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 529.211972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 529.217801] Lustre: Skipped 1 previous similar message [ 529.342845] Lustre: lustre-MDT0001: new disk, initializing [ 529.421360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 529.465129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 529.490370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 534.410618] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 539.278762] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 548.695591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 548.870072] Lustre: lustre-OST0000: new disk, initializing [ 548.872463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 548.877034] Lustre: 21189:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 548.931751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.750278] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 556.564640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 556.571687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 556.646669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 570.486140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 570.614275] Lustre: lustre-OST0001: new disk, initializing [ 570.617489] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 570.623395] Lustre: 22213:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 570.695171] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 577.137725] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 577.580299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 577.583295] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 577.643623] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 588.042880] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 600.175437] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 609.131675] Lustre: DEBUG MARKER: === sanity-scrub: finish setup 15:03:14 (1787338994) === [ 612.017541] Lustre: DEBUG MARKER: == sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 15:03:16 (1787338996) [ 635.861788] Lustre: Failing over lustre-MDT0000 [ 636.131744] Lustre: server umount lustre-MDT0000 complete [ 636.898375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 636.908707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.946572] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.964623] Lustre: Skipped 1 previous similar message [ 639.915361] LustreError: 22981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339027 with bad export cookie 12775008891202508501 [ 639.919111] Lustre: Failing over lustre-MDT0001 [ 639.923917] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.930871] LustreError: 22981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 640.587340] Lustre: server umount lustre-MDT0001 complete [ 649.922683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 650.209847] LustreError: 16397:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e2cdc47800 x1874160595897344/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 650.535115] LustreError: 21182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.551933] LustreError: 21182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 650.600174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.645397] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 655.237940] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 655.842882] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.845642] LustreError: 21183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.863795] Lustre: Skipped 1 previous similar message [ 655.887678] LustreError: 21183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 658.985798] LustreError: 24234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.996076] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 659.938202] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787339031/real 1787339031] req@ffff97e2ff83d880 x1874160595896960/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787339047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 664.045959] LustreError: 21182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.082032] LustreError: 21182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 665.056126] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787339036/real 1787339036] req@ffff97e2fa2d1880 x1874160595897088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787339052 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 665.101321] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 666.080330] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787339037/real 1787339037] req@ffff97e2cdc47480 x1874160595897472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787339053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 666.671252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 667.060419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 667.191140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 667.240881] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:42 to 0x2c0000400:65) [ 667.255702] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:65) [ 672.235932] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 672.247679] Lustre: Skipped 1 previous similar message [ 672.262213] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 672.291365] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 672.374123] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:65) [ 672.375114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 672.395338] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 686.980081] Lustre: DEBUG MARKER: == sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 15:04:32 (1787339072) [ 710.312418] Lustre: Failing over lustre-MDT0000 [ 710.597621] Lustre: server umount lustre-MDT0000 complete [ 713.184975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 713.188245] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.213393] LustreError: 24677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.220321] Lustre: Skipped 3 previous similar messages [ 713.238993] LustreError: 24677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 714.453074] LustreError: 19244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339101 with bad export cookie 12775008891202524825 [ 714.464415] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.466423] Lustre: Failing over lustre-MDT0001 [ 714.952069] Lustre: server umount lustre-MDT0001 complete [ 724.276269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 724.449601] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb149f2e6d295c191 [ 724.460759] Lustre: MGC192.168.202.108@tcp: Connection restored to 0@lo (at 0@lo) [ 724.474897] Lustre: Skipped 2 previous similar messages [ 724.799197] LustreError: 21182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 724.906359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 724.957285] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 729.264541] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 730.090501] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.102267] Lustre: Skipped 2 previous similar messages [ 733.220646] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 734.244082] Lustre: 16398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787339105/real 1787339105] req@ffff97e2ff7b8a80 x1874160596015360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787339121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 734.270617] Lustre: 16398:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 737.823793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 738.138731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 738.305991] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:105 to 0x280000400:129) [ 738.307231] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:106 to 0x2c0000400:129) [ 739.233245] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787339110/real 1787339110] req@ffff97e2d44e6680 x1874160596015744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787339126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 739.268933] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 743.407906] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 743.408718] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 743.421192] Lustre: Skipped 1 previous similar message [ 743.461185] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 743.520160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:129) [ 743.526804] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 743.561075] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 748.536696] Lustre: *** cfs_fail_loc=193, val=0*** [ 754.996239] Lustre: Failing over lustre-MDT0000 [ 755.281330] Lustre: server umount lustre-MDT0000 complete [ 758.757618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 758.763961] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 758.776794] LustreError: 21183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 758.776807] LustreError: 21183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 758.832883] Lustre: Skipped 3 previous similar messages [ 764.993825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 765.150420] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 765.405144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 765.409657] Lustre: Skipped 1 previous similar message [ 765.438949] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 770.539229] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 770.550532] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 770.557155] Lustre: Skipped 2 previous similar messages [ 770.571535] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 770.615165] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:161) [ 770.620645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:161) [ 770.679680] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 779.515615] Lustre: DEBUG MARKER: == sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 15:06:05 (1787339165) [ 792.055516] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 809.630277] Lustre: 30512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 831.851943] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 835.174445] Lustre: 31647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 848.735910] Lustre: *** cfs_fail_loc=198, val=0*** [ 860.800915] Lustre: Failing over lustre-MDT0000 [ 861.010197] Lustre: server umount lustre-MDT0000 complete [ 862.690297] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.693904] LustreError: 26635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 862.697088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 862.720276] Lustre: Skipped 3 previous similar messages [ 862.778272] LustreError: 26635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 865.537526] Lustre: Failing over lustre-MDT0001 [ 865.538984] LustreError: 19244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339252 with bad export cookie 12775008891202553784 [ 865.539698] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.553505] LustreError: 19244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 865.920280] Lustre: server umount lustre-MDT0001 complete [ 871.292968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 877.524094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 883.168519] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787339254/real 1787339254] req@ffff97e2fac11f80 x1874160596187136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787339270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 883.197147] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 887.337907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 890.400787] LustreError: 16397:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e2c6eec000 x1874160596189312/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 890.724414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.757627] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 895.192928] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 902.116645] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 902.123864] Lustre: Skipped 3 previous similar messages [ 904.561267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 904.796715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 904.992506] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:193) [ 905.037955] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:193) [ 910.114486] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 910.339479] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 910.368967] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 910.428475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:202 to 0x280000401:225) [ 910.432529] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:202 to 0x2c0000401:225) [ 926.615147] Lustre: DEBUG MARKER: == sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 15:08:31 (1787339311) [ 950.209634] Lustre: Failing over lustre-MDT0000 [ 950.538078] Lustre: server umount lustre-MDT0000 complete [ 951.268626] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.295223] Lustre: Skipped 4 previous similar messages [ 951.298724] LustreError: 33122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.311976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 951.336972] LustreError: 33122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 955.026252] LustreError: 19242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339342 with bad export cookie 12775008891202581189 [ 955.029664] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 955.032662] Lustre: Failing over lustre-MDT0001 [ 955.040170] LustreError: 19242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 955.456517] Lustre: server umount lustre-MDT0001 complete [ 960.503664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 965.494661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 972.710601] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787339343/real 1787339343] req@ffff97e2fda59880 x1874160596309632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787339359 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 972.740852] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 12 previous similar messages [ 974.510226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 974.562665] Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64 [ 979.945218] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb149f2e6d2969edc [ 979.956269] Lustre: MGC192.168.202.108@tcp: Connection restored to 0@lo (at 0@lo) [ 979.968197] Lustre: Skipped 4 previous similar messages [ 980.396255] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 985.384944] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 994.647673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 994.682580] Lustre: lustre-MDT0001: invalid oi count 63, remove them, then set it to 64 [ 994.990445] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:233 to 0x2c0000400:257) [ 994.992521] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:234 to 0x280000400:257) [ 999.866231] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1000.440584] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1000.530968] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1000.615512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 1000.616038] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:266 to 0x2c0000401:289) [ 1024.477254] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1043.005667] Lustre: 38948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1069.551699] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1073.314076] Lustre: 40085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1096.903587] Lustre: Failing over lustre-MDT0000 [ 1097.165789] Lustre: server umount lustre-MDT0000 complete [ 1097.703603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.713844] Lustre: Skipped 4 previous similar messages [ 1097.714159] LustreError: 36000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1097.714221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1097.714225] LustreError: Skipped 1 previous similar message [ 1097.753817] LustreError: 36000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 1101.476938] LustreError: 19244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339488 with bad export cookie 12775008891202608860 [ 1101.477327] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1101.477600] Lustre: Failing over lustre-MDT0001 [ 1101.495039] LustreError: 19244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1101.874569] Lustre: server umount lustre-MDT0001 complete [ 1106.643537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1114.504704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1118.176128] Lustre: 16398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787339489/real 1787339489] req@ffff97e2c4098380 x1874160596457984/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787339505 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1118.210480] Lustre: 16398:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 1127.273847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1127.322485] Lustre: lustre-MDT0000: invalid oi count 58, remove them, then set it to 64 [ 1145.888575] LustreError: 16397:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e2fc236680 x1874160596461056/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1146.358994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.367960] Lustre: Skipped 3 previous similar messages [ 1146.407528] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1151.709145] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1161.273406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1161.319429] Lustre: lustre-MDT0001: invalid oi count 58, remove them, then set it to 64 [ 1161.875768] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:297 to 0x2c0000400:321) [ 1161.876394] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:298 to 0x280000400:321) [ 1162.789663] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1162.811255] Lustre: Skipped 5 previous similar messages [ 1162.833963] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1162.865742] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1162.953662] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:330 to 0x2c0000401:353) [ 1162.971713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:329 to 0x280000401:353) [ 1166.525821] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1191.044772] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1211.956778] Lustre: 44794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1236.428435] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1241.086422] Lustre: 45931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1268.193261] Lustre: Failing over lustre-MDT0000 [ 1268.862942] Lustre: server umount lustre-MDT0000 complete [ 1270.752794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1270.768542] LustreError: Skipped 1 previous similar message [ 1270.772757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.785916] Lustre: Skipped 3 previous similar messages [ 1273.139351] LustreError: 25762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339660 with bad export cookie 12775008891202636349 [ 1273.148435] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.149704] Lustre: Failing over lustre-MDT0001 [ 1273.166959] LustreError: 25762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1273.459272] Lustre: server umount lustre-MDT0001 complete [ 1278.752230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1286.204569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1293.280745] Lustre: 16398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787339664/real 1787339664] req@ffff97e2c440df80 x1874160596616192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787339680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1293.307559] Lustre: 16398:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 12 previous similar messages [ 1297.326870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1297.400701] Lustre: lustre-MDT0000: invalid oi count 56, remove them, then set it to 64 [ 1298.470395] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb149f2e6d297759e [ 1298.480429] Lustre: MGC192.168.202.108@tcp: Connection restored to 0@lo (at 0@lo) [ 1298.482822] Lustre: Skipped 4 previous similar messages [ 1299.114256] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1303.655983] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1313.450501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1313.501835] Lustre: lustre-MDT0001: invalid oi count 56, remove them, then set it to 64 [ 1314.044113] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:362 to 0x2c0000400:385) [ 1314.048215] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:361 to 0x280000400:385) [ 1319.088230] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1319.413206] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1319.451123] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1319.506695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:394 to 0x280000401:417) [ 1319.512238] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:393 to 0x2c0000401:417) [ 1335.558446] Lustre: DEBUG MARKER: == sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 15:15:21 (1787339721) [ 1351.498942] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1373.515642] Lustre: 50644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1398.426910] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1425.009410] Lustre: Failing over lustre-MDT0000 [ 1425.347872] Lustre: server umount lustre-MDT0000 complete [ 1426.917406] LustreError: 21183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.945025] LustreError: 21183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 1429.244194] LustreError: 22981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339816 with bad export cookie 12775008891202663838 [ 1429.251652] Lustre: Failing over lustre-MDT0001 [ 1429.257822] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1429.261686] LustreError: 22981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1429.644986] Lustre: server umount lustre-MDT0001 complete [ 1436.406654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1448.422527] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787339819/real 1787339819] req@ffff97e1c23d9f80 x1874160596767744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787339835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1448.455141] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 1448.617553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1459.798801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1471.921862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1483.766594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1483.791643] Lustre: lustre-MDT0000: reset Object Index mappings [ 1500.422819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.429933] Lustre: Skipped 3 previous similar messages [ 1500.533640] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1507.177623] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1517.560797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1517.570889] Lustre: lustre-MDT0001: reset Object Index mappings [ 1517.846583] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:426 to 0x2c0000400:449) [ 1517.860148] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:425 to 0x280000400:449) [ 1518.852582] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1518.939695] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1518.997237] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:457 to 0x2c0000401:481) [ 1518.997368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:458 to 0x280000401:481) [ 1521.777320] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1535.700979] Lustre: DEBUG MARKER: == sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 15:18:41 (1787339921) [ 1557.628460] Lustre: Failing over lustre-MDT0000 [ 1557.965617] Lustre: server umount lustre-MDT0000 complete [ 1560.034651] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.036526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1560.052922] Lustre: Skipped 13 previous similar messages [ 1560.067110] LustreError: Skipped 2 previous similar messages [ 1562.488437] LustreError: 25762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787339949 with bad export cookie 12775008891202691327 [ 1562.492876] Lustre: Failing over lustre-MDT0001 [ 1562.495620] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1562.498532] LustreError: 25762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1562.933462] Lustre: server umount lustre-MDT0001 complete [ 1569.744469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1581.108449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1590.698371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1601.509894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1613.806436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1613.828933] Lustre: lustre-MDT0000: reset Object Index mappings [ 1632.736713] LustreError: 16397:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e2eb18b100 x1874160596894464/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1633.389934] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1638.769072] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1648.109580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1648.133055] Lustre: lustre-MDT0001: reset Object Index mappings [ 1648.606696] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:490 to 0x280000400:513) [ 1648.610909] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:489 to 0x2c0000400:513) [ 1650.619043] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1650.621669] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1650.642107] Lustre: Skipped 10 previous similar messages [ 1650.655993] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1650.736588] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:522 to 0x2c0000401:545) [ 1650.738683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:521 to 0x280000401:545) [ 1652.784445] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1661.406405] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004282:0x1:0x0]/64029: rc = 0 [ 1662.608276] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240003ab1:0x1:0x0]/52 with flags 0x52: rc = 0 [ 1690.344349] Lustre: DEBUG MARKER: == sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 15:21:14 (1787340074) [ 1719.563934] Lustre: Failing over lustre-MDT0000 [ 1719.867900] Lustre: server umount lustre-MDT0000 complete [ 1723.511723] LustreError: 19244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787340110 with bad export cookie 12775008891202718914 [ 1723.513789] Lustre: Failing over lustre-MDT0001 [ 1723.514500] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1723.535959] LustreError: 19244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1724.071245] Lustre: server umount lustre-MDT0001 complete [ 1729.457428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1739.700150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1744.672221] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787340115/real 1787340115] req@ffff97e2fd3ab800 x1874160597027840/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787340131 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1744.706891] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 24 previous similar messages [ 1747.364200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1757.450971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1769.528142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1769.557501] Lustre: lustre-MDT0000: reset Object Index mappings [ 1793.895090] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb149f2e6d298b903 [ 1794.420212] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1799.307397] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1808.439328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1808.469440] Lustre: lustre-MDT0001: reset Object Index mappings [ 1809.046431] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:554 to 0x2c0000400:577) [ 1809.049429] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:553 to 0x280000400:577) [ 1813.029233] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1813.109985] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1813.159417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:586 to 0x280000401:609) [ 1813.159558] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:585 to 0x2c0000401:609) [ 1813.853084] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1822.570149] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004a52:0x1:0x0]/32007: rc = 0 [ 1825.824874] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004281:0x1:0x0]/32003 with flags 0x52: rc = 0 [ 1948.090390] Lustre: DEBUG MARKER: == sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ========================================================== 15:25:33 (1787340333) [ 1984.691277] Lustre: Failing over lustre-MDT0000 [ 1985.110212] Lustre: server umount lustre-MDT0000 complete [ 1987.553614] LustreError: 21182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.585210] LustreError: 21182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [ 1989.599444] LustreError: 65693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787340376 with bad export cookie 12775008891202746627 [ 1989.610066] Lustre: Failing over lustre-MDT0001 [ 1989.610210] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1989.616897] LustreError: 65693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1990.084885] Lustre: server umount lustre-MDT0001 complete [ 1997.770530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2009.700523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2019.849855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2031.758373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2046.677183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2046.728273] Lustre: lustre-MDT0000: reset Object Index mappings [ 2059.232907] LustreError: 16397:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e2fadfed80 x1874160597222784/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2059.871037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.878834] Lustre: Skipped 5 previous similar messages [ 2059.922289] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2065.371779] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2073.366777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2073.596444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2073.605105] LustreError: Skipped 4 previous similar messages [ 2073.727274] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:617 to 0x280000400:641) [ 2073.727521] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:618 to 0x2c0000400:641) [ 2078.700115] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2078.744659] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2078.807771] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2078.810834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:650 to 0x280000401:673) [ 2078.818528] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:649 to 0x2c0000401:673) [ 2087.659801] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200005222:0x1:0x0]/32045: rc = 0 [ 2090.900464] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004a51:0x1:0x0]/32025 with flags 0x52: rc = 0 [ 2183.302695] Lustre: DEBUG MARKER: == sanity-scrub test 4d: FID in LMA mismatch with object FID won't block create ========================================================== 15:29:28 (1787340568) [ 2201.365334] Lustre: *** cfs_fail_loc=19b, val=0*** [ 2201.374041] Lustre: Skipped 1 previous similar message [ 2201.865154] Lustre: *** cfs_fail_loc=19b, val=0*** [ 2201.867064] Lustre: Skipped 34 previous similar messages [ 2202.887729] Lustre: *** cfs_fail_loc=19b, val=0*** [ 2202.892729] Lustre: Skipped 208 previous similar messages [ 2224.564895] Lustre: DEBUG MARKER: == sanity-scrub test 4e: FID reuse can be fixed ========== 15:30:09 (1787340609) [ 2229.324793] Lustre: *** cfs_fail_loc=1a0, val=0*** [ 2229.327850] Lustre: Skipped 211 previous similar messages [ 2245.715810] Lustre: DEBUG MARKER: == sanity-scrub test 5: OI scrub state machine =========== 15:30:31 (1787340631) [ 2273.256096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.263278] Lustre: Skipped 15 previous similar messages [ 2273.272103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.281685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.295225] Lustre: Skipped 3 previous similar messages [ 2278.903934] Lustre: server umount lustre-MDT0000 complete [ 2282.417063] LustreError: 22981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787340669 with bad export cookie 12775008891202790818 [ 2282.426864] LustreError: 22981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2284.517716] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2284.524687] Lustre: Skipped 1 previous similar message [ 2287.520995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2287.525975] Lustre: Skipped 1 previous similar message [ 2288.992851] Lustre: server umount lustre-MDT0001 complete [ 2293.259907] LustreError: 72740:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x317:0x0].0x0 (ffff97e2c5514e00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2299.463871] Lustre: server umount lustre-OST0000 complete [ 2318.304177] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2318.701160] Lustre: server umount lustre-OST0001 complete [ 2326.675783] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 2335.480891] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2386.219343] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2398.423913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.772796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2398.834260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2398.923519] Lustre: lustre-MDT0000: new disk, initializing [ 2399.054554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2404.261253] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2416.172779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2416.326220] Lustre: 75865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2416.341206] Lustre: 75865:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2416.385283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2416.389469] Lustre: Skipped 1 previous similar message [ 2416.457824] Lustre: lustre-MDT0001: new disk, initializing [ 2416.601151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2416.618369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2421.958927] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2427.790363] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2434.959660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2435.264800] Lustre: lustre-OST0000: new disk, initializing [ 2435.270380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2435.280334] Lustre: 77498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2437.162832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2437.172503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2437.263368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2441.663301] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2451.647909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2451.807286] Lustre: lustre-OST0001: new disk, initializing [ 2451.810863] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2451.819224] Lustre: 78370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2453.185221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2453.191635] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2453.258580] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2459.488472] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2469.392486] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2473.266678] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2500.751088] Lustre: Failing over lustre-MDT0000 [ 2501.324517] Lustre: server umount lustre-MDT0000 complete [ 2505.517678] Lustre: Failing over lustre-MDT0001 [ 2505.519234] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2505.526495] LustreError: Skipped 1 previous similar message [ 2506.044660] Lustre: server umount lustre-MDT0001 complete [ 2512.515087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2523.588891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2525.158602] Lustre: 16398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787340896/real 1787340896] req@ffff97e2fd112d80 x1874160597550848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787340912 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2525.179609] Lustre: 16398:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [ 2533.654935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2546.617352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2558.804257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2558.826523] Lustre: lustre-MDT0000: reset Object Index mappings [ 2558.830789] Lustre: Skipped 1 previous similar message [ 2575.392626] LustreError: 16397:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e2fadffb80 x1874160597554560/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2575.890429] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2580.947484] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2590.186530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2590.724961] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:65) [ 2590.730369] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:65) [ 2594.822131] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2594.839050] Lustre: Skipped 15 previous similar messages [ 2594.881995] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2594.936856] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2595.016112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 2595.023111] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:65) [ 2595.748245] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2606.298924] Lustre: *** cfs_fail_loc=190, val=3*** [ 2606.300608] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000402:0x1:0x0]/64029: rc = 0 [ 2607.387572] Lustre: *** cfs_fail_loc=190, val=3*** [ 2607.392208] Lustre: Skipped 1 previous similar message [ 2608.454745] Lustre: *** cfs_fail_loc=190, val=3*** [ 2609.619031] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240000402:0x1:0x0]/32007 with flags 0x52: rc = 0 [ 2611.488920] Lustre: *** cfs_fail_loc=190, val=3*** [ 2611.498842] Lustre: Skipped 2 previous similar messages [ 2615.714426] Lustre: *** cfs_fail_loc=190, val=3*** [ 2615.720409] Lustre: Skipped 2 previous similar messages [ 2623.088469] Lustre: Failing over lustre-MDT0000 [ 2623.359138] Lustre: server umount lustre-MDT0000 complete [ 2626.017699] LustreError: 82682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.031641] LustreError: 82682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 2627.214210] Lustre: Failing over lustre-MDT0001 [ 2627.483223] Lustre: server umount lustre-MDT0001 complete [ 2637.190432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2657.138276] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2665.783025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2666.324766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2666.328073] Lustre: Skipped 8 previous similar messages [ 2666.357173] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:97) [ 2666.365844] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:97) [ 2667.401197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:97) [ 2667.404925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:97) [ 2671.320176] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2677.291787] Lustre: Failing over lustre-MDT0000 [ 2677.673820] Lustre: server umount lustre-MDT0000 complete [ 2682.323852] Lustre: Failing over lustre-MDT0001 [ 2682.789664] Lustre: server umount lustre-MDT0001 complete [ 2693.688046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2693.826042] Lustre: *** cfs_fail_loc=190, val=3*** [ 2693.831094] Lustre: Skipped 2 previous similar messages [ 2707.232894] LustreError: 16397:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e2f689bb80 x1874160597620224/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2711.968107] Lustre: *** cfs_fail_loc=190, val=3*** [ 2711.970574] Lustre: Skipped 5 previous similar messages [ 2713.332499] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2724.140286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2724.478641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2724.496857] LustreError: Skipped 5 previous similar messages [ 2724.719170] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:129) [ 2724.731084] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:129) [ 2730.089916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:129) [ 2730.090031] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:129) [ 2730.507388] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2744.860670] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240000402:0x1:0x0]/32007 with flags 0x52: rc = 0 [ 2744.881699] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000402:0x45:0x0]/285: rc = 0 [ 2758.465989] Lustre: DEBUG MARKER: == sanity-scrub test 6: OI scrub resumes from last checkpoint ========================================================== 15:39:04 (1787341144) [ 2784.613518] Lustre: Failing over lustre-MDT0000 [ 2784.932824] Lustre: server umount lustre-MDT0000 complete [ 2788.543534] Lustre: Failing over lustre-MDT0001 [ 2788.888727] Lustre: server umount lustre-MDT0001 complete [ 2794.657649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2805.250602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2814.644038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2828.394350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2846.273090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2846.293223] Lustre: lustre-MDT0000: reset Object Index mappings [ 2846.296478] Lustre: Skipped 1 previous similar message [ 2863.210352] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2870.794310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2871.235713] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:193) [ 2871.253861] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:169 to 0x2c0000400:193) [ 2876.345923] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2876.448201] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:169 to 0x2c0000401:193) [ 2876.455136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2886.335107] Lustre: *** cfs_fail_loc=190, val=2*** [ 2886.338765] Lustre: Skipped 15 previous similar messages [ 2886.343040] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200001b72:0x1:0x0]/32004: rc = 0 [ 2889.635215] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240001b71:0x1:0x0]/47 with flags 0x52: rc = 0 [ 2902.762223] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200001b72:0x45:0x0]/206: rc = 0 [ 2917.978473] Lustre: Failing over lustre-MDT0000 [ 2918.371935] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.389970] Lustre: Skipped 32 previous similar messages [ 2918.481927] Lustre: server umount lustre-MDT0000 complete [ 2922.616511] Lustre: Failing over lustre-MDT0001 [ 2922.619250] LustreError: 75858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787341309 with bad export cookie 12775008891202948640 [ 2922.633299] LustreError: 75858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 17 previous similar messages [ 2922.870730] Lustre: server umount lustre-MDT0001 complete [ 2933.515915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2951.715173] Lustre: *** cfs_fail_loc=190, val=3*** [ 2951.719415] Lustre: Skipped 30 previous similar messages [ 2953.347814] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2963.514771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2963.885093] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:169 to 0x2c0000400:225) [ 2963.888851] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:225) [ 2968.724573] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2969.127491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:225) [ 2969.135221] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:169 to 0x2c0000401:225) [ 2986.417926] Lustre: DEBUG MARKER: == sanity-scrub test 7: System is available during OI scrub scanning ========================================================== 15:42:52 (1787341372) [ 3002.086791] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3023.664617] Lustre: 96520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3049.938146] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3054.146968] Lustre: 97657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3097.869061] Lustre: Failing over lustre-MDT0000 [ 3098.344173] Lustre: server umount lustre-MDT0000 complete [ 3102.741070] Lustre: Failing over lustre-MDT0001 [ 3103.250085] Lustre: server umount lustre-MDT0001 complete [ 3109.858506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3120.592970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3128.802399] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787341499/real 1787341499] req@ffff97e1cad77480 x1874160597951744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787341515 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3128.827502] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 52 previous similar messages [ 3129.728907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3140.560579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3153.004745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3153.049980] Lustre: lustre-MDT0000: reset Object Index mappings [ 3153.054607] Lustre: Skipped 1 previous similar message [ 3178.894447] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3187.477555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3188.045110] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:265 to 0x280000400:289) [ 3188.049745] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:266 to 0x2c0000400:289) [ 3192.224825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:266 to 0x280000401:289) [ 3192.224970] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:265 to 0x2c0000401:289) [ 3193.075146] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3202.665901] Lustre: *** cfs_fail_loc=190, val=3*** [ 3202.668271] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200002b12:0x1:0x0]/32014: rc = 0 [ 3202.669834] Lustre: Skipped 11 previous similar messages [ 3202.688695] Lustre: Skipped 1 previous similar message [ 3206.033192] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240002b11:0x1:0x0]/32033 with flags 0x52: rc = 0 [ 3229.297858] Lustre: DEBUG MARKER: == sanity-scrub test 8: Control OI scrub manually ======== 15:46:54 (1787341614) [ 3275.265982] Lustre: Failing over lustre-MDT0000 [ 3275.810304] Lustre: server umount lustre-MDT0000 complete [ 3279.338716] LustreError: 82681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.350167] LustreError: 82681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 59 previous similar messages [ 3279.353372] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3279.382165] LustreError: Skipped 5 previous similar messages [ 3279.396097] Lustre: Failing over lustre-MDT0001 [ 3279.768770] Lustre: server umount lustre-MDT0001 complete [ 3285.209334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3295.347430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3305.297789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3316.577950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3330.577966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3349.382784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3349.389818] Lustre: Skipped 8 previous similar messages [ 3349.429109] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3349.453431] Lustre: Skipped 5 previous similar messages [ 3354.534865] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3363.602633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3363.781616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 3363.787282] LustreError: Skipped 6 previous similar messages [ 3363.929516] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:329 to 0x2c0000400:353) [ 3363.934579] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:330 to 0x280000400:353) [ 3365.923031] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3365.930734] Lustre: Skipped 5 previous similar messages [ 3365.931260] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3365.946673] Lustre: Skipped 29 previous similar messages [ 3365.979967] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3365.985539] Lustre: Skipped 5 previous similar messages [ 3366.044966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:330 to 0x2c0000401:353) [ 3366.044966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:329 to 0x280000401:353) [ 3369.236745] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3409.027589] Lustre: DEBUG MARKER: == sanity-scrub test 9: OI scrub speed control =========== 15:49:54 (1787341794) [ 3426.224231] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3446.665489] Lustre: 109203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3469.537548] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3593.293285] Lustre: Failing over lustre-MDT0000 [ 3593.852572] Lustre: server umount lustre-MDT0000 complete [ 3596.277955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.288348] Lustre: Skipped 14 previous similar messages [ 3598.540831] Lustre: Failing over lustre-MDT0001 [ 3598.544917] LustreError: 75859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787341985 with bad export cookie 12775008891203090502 [ 3598.570552] LustreError: 75859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3599.264874] Lustre: server umount lustre-MDT0001 complete [ 3605.064816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3616.400377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3631.853738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3647.457064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3663.278497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3663.304965] Lustre: lustre-MDT0000: reset Object Index mappings [ 3663.308660] Lustre: Skipped 3 previous similar messages [ 3667.941167] LustreError: 16397:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e1ca7c0380 x1874160598300544/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3673.417415] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3682.226137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3682.609917] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:393 to 0x2c0000400:417) [ 3682.634639] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:394 to 0x280000400:417) [ 3686.736928] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:394 to 0x2c0000401:417) [ 3686.741277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:393 to 0x280000401:417) [ 3686.923801] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3751.853304] Lustre: DEBUG MARKER: == sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount (1) ========================================================== 15:55:37 (1787342137) [ 3768.623231] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3790.596887] Lustre: 117172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3790.610981] Lustre: 117172:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 3821.501479] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4038.605541] Lustre: Failing over lustre-MDT0000 [ 4039.004093] Lustre: server umount lustre-MDT0000 complete [ 4040.173053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4040.194268] LustreError: Skipped 1 previous similar message [ 4040.209148] LustreError: 82681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4040.231868] LustreError: 82681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 4043.423837] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4043.429176] LustreError: Skipped 1 previous similar message [ 4043.448160] Lustre: Failing over lustre-MDT0001 [ 4044.005849] Lustre: server umount lustre-MDT0001 complete [ 4051.622617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4062.562569] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787342433/real 1787342433] req@ffff97e2ea4e4380 x1874160598568064/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787342449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4062.615337] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 35 previous similar messages [ 4064.486860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4075.627039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4087.977139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4101.726578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4114.582258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4114.588298] Lustre: Skipped 3 previous similar messages [ 4114.630761] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4114.649449] Lustre: Skipped 1 previous similar message [ 4121.084065] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4131.299712] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4131.318912] Lustre: Skipped 9 previous similar messages [ 4133.167316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4133.795268] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:457 to 0x280000400:481) [ 4133.799376] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:458 to 0x2c0000400:481) [ 4138.980766] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4139.008357] Lustre: Skipped 1 previous similar message [ 4139.077397] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4139.104036] Lustre: Skipped 1 previous similar message [ 4139.187121] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:457 to 0x2c0000401:481) [ 4139.187929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:458 to 0x280000401:481) [ 4140.215641] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4152.128449] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004282:0x1:0x0]/32009: rc = 0 [ 4152.129643] Lustre: *** cfs_fail_loc=190, val=1*** [ 4152.138060] Lustre: Skipped 40 previous similar messages [ 4155.410868] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004281:0x1:0x0]/32006 with flags 0x52: rc = 0 [ 4166.870491] Lustre: Failing over lustre-MDT0000 [ 4167.859211] Lustre: server umount lustre-MDT0000 complete [ 4173.704729] Lustre: Failing over lustre-MDT0001 [ 4174.287819] Lustre: server umount lustre-MDT0001 complete [ 4184.886714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4198.305615] LustreError: 16397:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e1d0462d80 x1874160598611840/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4203.785462] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4212.924823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4213.290774] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:458 to 0x2c0000400:513) [ 4213.296995] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:457 to 0x280000400:513) [ 4218.046498] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4218.393176] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:457 to 0x2c0000401:513) [ 4218.396781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:458 to 0x280000401:513) [ 4224.391307] Lustre: Failing over lustre-MDT0000 [ 4224.732040] Lustre: server umount lustre-MDT0000 complete [ 4228.581475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.599784] Lustre: Skipped 16 previous similar messages [ 4228.727932] Lustre: Failing over lustre-MDT0001 [ 4228.729721] LustreError: 77508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787342615 with bad export cookie 12775008891204001741 [ 4228.744921] LustreError: 77508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4229.058508] Lustre: server umount lustre-MDT0001 complete [ 4238.688316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4238.838585] Lustre: *** cfs_fail_loc=190, val=1*** [ 4238.848107] Lustre: Skipped 36 previous similar messages [ 4259.252636] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4269.295730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4269.677157] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:457 to 0x280000400:545) [ 4269.685865] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:458 to 0x2c0000400:545) [ 4274.761748] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:457 to 0x2c0000401:545) [ 4274.766809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:458 to 0x280000401:545) [ 4275.182740] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4292.293688] Lustre: DEBUG MARKER: == sanity-scrub test 11: OI scrub skips the new created objects only once ========================================================== 16:04:38 (1787342678) [ 4309.601988] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 4331.785775] Lustre: 128217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4331.805060] Lustre: 128217:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 4361.600783] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4400.940863] Lustre: DEBUG MARKER: == sanity-scrub test 12: OI scrub can rebuild invalid /O entries ========================================================== 16:06:26 (1787342786) [ 4406.758916] Lustre: *** cfs_fail_loc=195, val=0*** [ 4411.193636] Lustre: Failing over lustre-OST0000 [ 4411.284147] Lustre: server umount lustre-OST0000 complete [ 4422.402447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4431.883194] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4623.626529] Lustre: DEBUG MARKER: == sanity-scrub test 13: OI scrub can rebuild missed /O entries ========================================================== 16:10:09 (1787343009) [ 4628.171155] Lustre: *** cfs_fail_loc=196, val=0*** [ 4628.178030] Lustre: Skipped 63 previous similar messages [ 4632.828527] Lustre: Failing over lustre-OST0000 [ 4632.944203] Lustre: server umount lustre-OST0000 complete [ 4641.352902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4641.536902] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4647.541465] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4836.393334] Lustre: DEBUG MARKER: == sanity-scrub test 14: OI scrub can repair OST objects under lost+found ========================================================== 16:13:42 (1787343222) [ 4843.558389] Lustre: *** cfs_fail_loc=196, val=0*** [ 4843.560652] Lustre: Skipped 63 previous similar messages [ 4845.823711] Lustre: *** cfs_fail_loc=196, val=0*** [ 4845.832594] Lustre: Skipped 191 previous similar messages [ 4850.061291] Lustre: *** cfs_fail_loc=196, val=0*** [ 4850.066743] Lustre: Skipped 351 previous similar messages [ 4865.709639] Lustre: Failing over lustre-OST0000 [ 4865.857550] Lustre: server umount lustre-OST0000 complete [ 4867.061153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4867.076334] Lustre: Skipped 8 previous similar messages [ 4867.086196] LustreError: 97679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4867.102293] LustreError: 97679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 50 previous similar messages [ 4877.018091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4877.281157] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4877.289677] Lustre: Skipped 1 previous similar message [ 4877.367921] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4877.388655] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4877.401357] Lustre: Skipped 4 previous similar messages [ 4878.945580] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4878.957150] Lustre: Skipped 4 previous similar messages [ 4878.977036] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4878.977897] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4878.990140] Lustre: Skipped 4 previous similar messages [ 4879.003432] Lustre: Skipped 18 previous similar messages [ 4884.128152] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4897.771874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4897.784470] Lustre: Skipped 3 previous similar messages [ 4897.897830] Lustre: server umount lustre-MDT0000 complete [ 4902.000964] LustreError: 88079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787343289 with bad export cookie 12775008891204003372 [ 4902.018167] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4902.031034] LustreError: 88079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4902.076958] LustreError: Skipped 2 previous similar messages [ 4902.537626] Lustre: server umount lustre-MDT0001 complete [ 4916.212484] Lustre: server umount lustre-OST0000 complete [ 4919.265586] Lustre: 16398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787343290/real 1787343290] req@ffff97e2feb55880 x1874160599073280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787343306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4919.313436] Lustre: 16398:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 33 previous similar messages [ 4920.576523] Lustre: server umount lustre-OST0001 complete [ 4931.706847] Lustre: DEBUG MARKER: == sanity-scrub test 15: Dryrun mode OI scrub ============ 16:15:17 (1787343317) [ 4950.084268] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 4961.152533] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 5013.449645] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 5024.580668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5024.804718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5024.850745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5024.940645] Lustre: lustre-MDT0000: new disk, initializing [ 5025.037871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5025.043706] Lustre: Skipped 6 previous similar messages [ 5025.058892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5030.227827] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5041.347445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5041.463949] Lustre: 139537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5041.479606] Lustre: 139537:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 5041.518800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5041.524221] Lustre: Skipped 1 previous similar message [ 5041.621269] Lustre: lustre-MDT0001: new disk, initializing [ 5041.732339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5041.736142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5046.346771] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5051.927374] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5060.158131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5060.431531] Lustre: lustre-OST0000: new disk, initializing [ 5060.439367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5060.446652] Lustre: 141166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5061.927435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5061.934810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5061.981169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5067.090942] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5078.105592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5078.222467] Lustre: lustre-OST0001: new disk, initializing [ 5078.224856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5078.228674] Lustre: 142037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5079.760364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5079.770222] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5079.886265] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5084.299891] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5095.991157] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5100.721379] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5125.406520] Lustre: Failing over lustre-MDT0000 [ 5125.726969] Lustre: server umount lustre-MDT0000 complete [ 5128.672630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5128.680977] LustreError: Skipped 5 previous similar messages [ 5130.125682] Lustre: Failing over lustre-MDT0001 [ 5130.526770] Lustre: server umount lustre-MDT0001 complete [ 5136.993947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5148.725940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5158.229128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5169.704487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5182.121566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5182.166117] Lustre: lustre-MDT0000: reset Object Index mappings [ 5182.169339] Lustre: Skipped 3 previous similar messages [ 5200.864449] LustreError: 16397:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e2eaa1d180 x1874160599198848/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5206.768575] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5217.005271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5217.476326] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:65) [ 5217.477134] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:42 to 0x2c0000400:65) [ 5219.569760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 5219.569915] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:65) [ 5221.500697] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5277.656920] Lustre: DEBUG MARKER: == sanity-scrub test 16: Initial OI scrub can rebuild crashed index objects ========================================================== 16:21:03 (1787343663) [ 5295.614374] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 5340.561377] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5373.780612] Lustre: Failing over lustre-MDT0000 [ 5373.806312] Lustre: *** cfs_fail_loc=199, val=0*** [ 5373.813213] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000001:0x3:0x0]: rc = 0 [ 5373.831773] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2:0x0]: rc = 0 [ 5373.840651] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x9:0x0]: rc = 0 [ 5373.849479] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xa:0x0]: rc = 0 [ 5373.857882] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xb:0x0]: rc = 0 [ 5373.867702] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xc:0x0]: rc = 0 [ 5373.875193] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xd:0x0]: rc = 0 [ 5373.881984] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xe:0x0]: rc = 0 [ 5373.891412] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x13:0x0]: rc = 0 [ 5373.899360] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x14:0x0]: rc = 0 [ 5373.910748] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x15:0x0]: rc = 0 [ 5373.929691] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x16:0x0]: rc = 0 [ 5373.944251] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x17:0x0]: rc = 0 [ 5373.959584] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x18:0x0]: rc = 0 [ 5373.974523] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x19:0x0]: rc = 0 [ 5373.990856] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1a:0x0]: rc = 0 [ 5374.001328] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1b:0x0]: rc = 0 [ 5374.015945] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1c:0x0]: rc = 0 [ 5374.029627] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1d:0x0]: rc = 0 [ 5374.039712] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1e:0x0]: rc = 0 [ 5374.054760] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1f:0x0]: rc = 0 [ 5374.077993] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x20:0x0]: rc = 0 [ 5374.095663] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x21:0x0]: rc = 0 [ 5374.115599] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x22:0x0]: rc = 0 [ 5374.126819] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x23:0x0]: rc = 0 [ 5374.144822] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x25:0x0]: rc = 0 [ 5374.158653] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x26:0x0]: rc = 0 [ 5374.171916] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x27:0x0]: rc = 0 [ 5374.195601] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x28:0x0]: rc = 0 [ 5374.219996] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x29:0x0]: rc = 0 [ 5374.250799] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2a:0x0]: rc = 0 [ 5374.270776] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2b:0x0]: rc = 0 [ 5374.298610] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2c:0x0]: rc = 0 [ 5374.311380] Lustre: *** cfs_fail_loc=199, val=0*** [ 5374.320229] Lustre: Skipped 32 previous similar messages [ 5374.326795] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2d:0x0]: rc = 0 [ 5374.338378] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2e:0x0]: rc = 0 [ 5374.350359] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2f:0x0]: rc = 0 [ 5374.359955] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x30:0x0]: rc = 0 [ 5374.375849] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x31:0x0]: rc = 0 [ 5374.392123] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x32:0x0]: rc = 0 [ 5374.402235] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x33:0x0]: rc = 0 [ 5374.418389] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x34:0x0]: rc = 0 [ 5374.428308] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x35:0x0]: rc = 0 [ 5374.438621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5374.442902] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x1:0x0]: rc = 0 [ 5374.445769] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x2:0x0]: rc = 0 [ 5374.454417] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x3:0x0]: rc = 0 [ 5374.459942] Lustre: Skipped 3 previous similar messages [ 5374.477577] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20001:0x0]: rc = 0 [ 5374.485630] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20002:0x0]: rc = 0 [ 5374.497249] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20003:0x0]: rc = 0 [ 5374.504224] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x10000:0x0]: rc = 0 [ 5374.517824] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x20000:0x0]: rc = 0 [ 5374.529740] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x1010000:0x0]: rc = 0 [ 5374.551946] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x1020000:0x0]: rc = 0 [ 5374.560895] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x2010000:0x0]: rc = 0 [ 5374.569499] Lustre: 151777:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x2020000:0x0]: rc = 0 [ 5376.992980] Lustre: server umount lustre-MDT0000 complete [ 5381.387542] Lustre: Failing over lustre-MDT0001 [ 5381.390862] Lustre: *** cfs_fail_loc=199, val=0*** [ 5381.392624] Lustre: Skipped 20 previous similar messages [ 5381.393905] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000001:0x3:0x0]: rc = 0 [ 5381.397197] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x3:0x0]: rc = 0 [ 5381.401147] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x4:0x0]: rc = 0 [ 5381.405153] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x5:0x0]: rc = 0 [ 5381.410080] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x6:0x0]: rc = 0 [ 5381.414905] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x7:0x0]: rc = 0 [ 5381.421262] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x8:0x0]: rc = 0 [ 5381.426535] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xd:0x0]: rc = 0 [ 5381.443544] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xe:0x0]: rc = 0 [ 5381.460735] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xf:0x0]: rc = 0 [ 5381.465192] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x10:0x0]: rc = 0 [ 5381.470304] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x11:0x0]: rc = 0 [ 5381.475277] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x12:0x0]: rc = 0 [ 5381.481601] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x13:0x0]: rc = 0 [ 5381.488884] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x14:0x0]: rc = 0 [ 5381.497500] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x15:0x0]: rc = 0 [ 5381.504869] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x16:0x0]: rc = 0 [ 5381.511425] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x17:0x0]: rc = 0 [ 5381.518400] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x18:0x0]: rc = 0 [ 5381.528870] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x19:0x0]: rc = 0 [ 5381.537223] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1a:0x0]: rc = 0 [ 5381.546585] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1b:0x0]: rc = 0 [ 5381.563261] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1c:0x0]: rc = 0 [ 5381.570316] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1d:0x0]: rc = 0 [ 5381.582251] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1f:0x0]: rc = 0 [ 5381.596519] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x20:0x0]: rc = 0 [ 5381.608079] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x21:0x0]: rc = 0 [ 5381.613995] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x22:0x0]: rc = 0 [ 5381.620688] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x23:0x0]: rc = 0 [ 5381.630128] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x24:0x0]: rc = 0 [ 5381.638232] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x25:0x0]: rc = 0 [ 5381.654951] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x26:0x0]: rc = 0 [ 5381.677336] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x27:0x0]: rc = 0 [ 5381.696784] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x28:0x0]: rc = 0 [ 5381.706951] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x29:0x0]: rc = 0 [ 5381.716193] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2a:0x0]: rc = 0 [ 5381.723789] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2b:0x0]: rc = 0 [ 5381.738975] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2c:0x0]: rc = 0 [ 5381.755736] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2d:0x0]: rc = 0 [ 5381.770490] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2e:0x0]: rc = 0 [ 5381.781837] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2f:0x0]: rc = 0 [ 5381.799491] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x1:0x0]: rc = 0 [ 5381.809728] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x2:0x0]: rc = 0 [ 5381.816128] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x3:0x0]: rc = 0 [ 5381.822947] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20001:0x0]: rc = 0 [ 5381.830626] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20002:0x0]: rc = 0 [ 5381.835772] Lustre: 151978:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20003:0x0]: rc = 0 [ 5382.141677] Lustre: server umount lustre-MDT0001 complete [ 5393.040533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5393.152774] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace' with [0x200000003:0x13:0x0]: rc = 0 [ 5393.170471] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5393.182286] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x2:0x0]: rc = 0 [ 5393.196529] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_13' with [0x200000003:0x21:0x0]: rc = 0 [ 5393.209743] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_00' with [0x200000003:0x25:0x0]: rc = 0 [ 5393.223746] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_07' with [0x200000003:0x1b:0x0]: rc = 0 [ 5393.242463] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_12' with [0x200000003:0x20:0x0]: rc = 0 [ 5393.256609] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_14' with [0x200000003:0x33:0x0]: rc = 0 [ 5393.273919] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_15' with [0x200000003:0x34:0x0]: rc = 0 [ 5393.295947] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_13' with [0x200000003:0x32:0x0]: rc = 0 [ 5393.317198] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_08' with [0x200000003:0x2d:0x0]: rc = 0 [ 5393.341800] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_05' with [0x200000003:0x19:0x0]: rc = 0 [ 5393.355479] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_03' with [0x200000003:0x28:0x0]: rc = 0 [ 5393.375990] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_01' with [0x200000003:0x26:0x0]: rc = 0 [ 5393.393544] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_06' with [0x200000003:0x2b:0x0]: rc = 0 [ 5393.407250] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_08' with [0x200000003:0x1c:0x0]: rc = 0 [ 5393.447563] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_14' with [0x200000003:0x22:0x0]: rc = 0 [ 5393.465586] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_03' with [0x200000003:0x17:0x0]: rc = 0 [ 5393.479486] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_02' with [0x200000003:0x27:0x0]: rc = 0 [ 5393.499644] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_04' with [0x200000003:0x29:0x0]: rc = 0 [ 5393.508204] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_04' with [0x200000003:0x18:0x0]: rc = 0 [ 5393.524222] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_10' with [0x200000003:0x2f:0x0]: rc = 0 [ 5393.544574] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_00' with [0x200000003:0x14:0x0]: rc = 0 [ 5393.554694] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_02' with [0x200000003:0x16:0x0]: rc = 0 [ 5393.569249] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_09' with [0x200000003:0x2e:0x0]: rc = 0 [ 5393.577753] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_11' with [0x200000003:0x30:0x0]: rc = 0 [ 5393.596374] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_01' with [0x200000003:0x15:0x0]: rc = 0 [ 5393.607920] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_15' with [0x200000003:0x23:0x0]: rc = 0 [ 5393.616406] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_12' with [0x200000003:0x31:0x0]: rc = 0 [ 5393.624709] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_06' with [0x200000003:0x1a:0x0]: rc = 0 [ 5393.634441] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_07' with [0x200000003:0x2c:0x0]: rc = 0 [ 5393.646094] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_11' with [0x200000003:0x1f:0x0]: rc = 0 [ 5393.655706] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_10' with [0x200000003:0x1e:0x0]: rc = 0 [ 5393.663252] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_05' with [0x200000003:0x2a:0x0]: rc = 0 [ 5393.681540] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_09' with [0x200000003:0x1d:0x0]: rc = 0 [ 5393.689971] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0x9:0x0]: rc = 0 [ 5393.701088] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000003:0xd:0x0]: rc = 0 [ 5393.713648] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5393.725317] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xb:0x0]: rc = 0 [ 5393.744611] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5393.755816] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xa:0x0]: rc = 0 [ 5393.764224] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000003:0xc:0x0]: rc = 0 [ 5393.772380] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000-MDT0000' with [0x200000005:0x20001:0x0]: rc = 0 [ 5393.793330] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000003:0xe:0x0]: rc = 0 [ 5393.804047] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5393.811936] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000-MDT0000' with [0x200000005:0x20002:0x0]: rc = 0 [ 5393.821043] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000-MDT0000' with [0x200000005:0x20003:0x0]: rc = 0 [ 5393.829370] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 5393.837166] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 5393.845806] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 5393.854599] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 5393.865838] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 5393.877779] Lustre: 152475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 5410.702187] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5419.724080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5419.913304] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace' with [0x200000003:0xd:0x0]: rc = 0 [ 5419.926903] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5419.934984] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x3:0x0]: rc = 0 [ 5419.943498] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x5:0x0]: rc = 0 [ 5419.974514] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5420.002417] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2020000' with [0x200000003:0x8:0x0]: rc = 0 [ 5420.027729] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5420.052535] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x4:0x0]: rc = 0 [ 5420.066318] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5420.082968] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x20000' with [0x200000003:0x6:0x0]: rc = 0 [ 5420.095259] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x20000-MDT0001' with [0x200000005:0x20001:0x0]: rc = 0 [ 5420.111566] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2020000-MDT0001' with [0x200000005:0x20003:0x0]: rc = 0 [ 5420.138934] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1020000-MDT0001' with [0x200000005:0x20002:0x0]: rc = 0 [ 5420.156402] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5420.179303] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_15' with [0x200000003:0x1d:0x0]: rc = 0 [ 5420.200719] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_03' with [0x200000003:0x11:0x0]: rc = 0 [ 5420.213339] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_12' with [0x200000003:0x2b:0x0]: rc = 0 [ 5420.233954] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_09' with [0x200000003:0x17:0x0]: rc = 0 [ 5420.246719] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_03' with [0x200000003:0x22:0x0]: rc = 0 [ 5420.263401] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_11' with [0x200000003:0x2a:0x0]: rc = 0 [ 5420.277894] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_07' with [0x200000003:0x15:0x0]: rc = 0 [ 5420.295093] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_13' with [0x200000003:0x1b:0x0]: rc = 0 [ 5420.304232] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_08' with [0x200000003:0x16:0x0]: rc = 0 [ 5420.315697] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_06' with [0x200000003:0x14:0x0]: rc = 0 [ 5420.327193] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_09' with [0x200000003:0x28:0x0]: rc = 0 [ 5420.344242] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_14' with [0x200000003:0x1c:0x0]: rc = 0 [ 5420.355863] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_01' with [0x200000003:0xf:0x0]: rc = 0 [ 5420.367416] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_05' with [0x200000003:0x13:0x0]: rc = 0 [ 5420.389407] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_00' with [0x200000003:0x1f:0x0]: rc = 0 [ 5420.424970] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_04' with [0x200000003:0x23:0x0]: rc = 0 [ 5420.450094] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_11' with [0x200000003:0x19:0x0]: rc = 0 [ 5420.466000] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_02' with [0x200000003:0x10:0x0]: rc = 0 [ 5420.488420] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_04' with [0x200000003:0x12:0x0]: rc = 0 [ 5420.514461] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_06' with [0x200000003:0x25:0x0]: rc = 0 [ 5420.543341] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_13' with [0x200000003:0x2c:0x0]: rc = 0 [ 5420.569283] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_02' with [0x200000003:0x21:0x0]: rc = 0 [ 5420.590969] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_15' with [0x200000003:0x2e:0x0]: rc = 0 [ 5420.610251] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_08' with [0x200000003:0x27:0x0]: rc = 0 [ 5420.628468] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_07' with [0x200000003:0x26:0x0]: rc = 0 [ 5420.654066] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_01' with [0x200000003:0x20:0x0]: rc = 0 [ 5420.673862] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_10' with [0x200000003:0x18:0x0]: rc = 0 [ 5420.696452] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_12' with [0x200000003:0x1a:0x0]: rc = 0 [ 5420.706975] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_14' with [0x200000003:0x2d:0x0]: rc = 0 [ 5420.716918] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_00' with [0x200000003:0xe:0x0]: rc = 0 [ 5420.737287] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_05' with [0x200000003:0x24:0x0]: rc = 0 [ 5420.760113] Lustre: 153222:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_10' with [0x200000003:0x29:0x0]: rc = 0 [ 5421.287398] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:105 to 0x2c0000400:129) [ 5421.300764] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:106 to 0x280000400:129) [ 5426.817272] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:106 to 0x2c0000401:129) [ 5426.818711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 5426.872138] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5442.970532] Lustre: DEBUG MARKER: == sanity-scrub test 17a: ENOSPC on OI insert shouldn't leak inodes ========================================================== 16:23:48 (1787343828) [ 5443.989795] Lustre: *** cfs_fail_loc=19d, val=0*** [ 5443.993898] Lustre: Skipped 123 previous similar messages [ 5445.950384] Lustre: Failing over lustre-MDT0000 [ 5446.288594] Lustre: server umount lustre-MDT0000 complete [ 5461.370299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5466.547891] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5467.230909] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:106 to 0x2c0000401:161) [ 5467.231136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:161) [ 5471.256818] Lustre: DEBUG MARKER: == sanity-scrub test 17b: ENOSPC on .. insertion shouldn't leak inodes ========================================================== 16:24:16 (1787343856) [ 5472.324771] Lustre: *** cfs_fail_loc=19e, val=0*** [ 5474.267410] Lustre: Failing over lustre-MDT0000 [ 5474.591243] Lustre: server umount lustre-MDT0000 complete [ 5476.424891] LustreError: 154627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.439687] LustreError: 154627:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 5477.346929] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5477.359356] Lustre: Skipped 25 previous similar messages [ 5490.123130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5490.701819] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5490.718347] Lustre: Skipped 3 previous similar messages [ 5491.773670] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5491.779182] Lustre: Skipped 3 previous similar messages [ 5495.732416] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5495.792455] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5495.799673] Lustre: Skipped 15 previous similar messages [ 5495.847850] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5495.855480] Lustre: Skipped 3 previous similar messages [ 5495.910342] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:106 to 0x2c0000401:193) [ 5495.917248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:193) [ 5499.527285] Lustre: DEBUG MARKER: == sanity-scrub test 18: test mount -o resetoi to recreate OI files ========================================================== 16:24:45 (1787343885) [ 5522.126236] Lustre: Failing over lustre-MDT0000 [ 5522.371752] Lustre: server umount lustre-MDT0000 complete [ 5526.128784] LustreError: 139529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787343913 with bad export cookie 12775008891204170812 [ 5526.135915] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5526.137939] LustreError: 139529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5526.142969] Lustre: Failing over lustre-MDT0001 [ 5526.163289] LustreError: Skipped 4 previous similar messages [ 5526.542834] Lustre: server umount lustre-MDT0001 complete [ 5536.588566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5542.880223] Lustre: 16398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787343913/real 1787343913] req@ffff97e2fcebc380 x1874160599535744/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787343929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5542.907597] Lustre: 16398:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 25 previous similar messages [ 5551.136681] LustreError: 16397:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e2c5c0a300 x1874160599538304/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5556.837376] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5566.932460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5567.627034] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:169 to 0x2c0000400:193) [ 5567.631936] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:193) [ 5571.439386] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5572.699922] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:257) [ 5572.702622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:234 to 0x280000401:257) [ 5578.036162] Lustre: Failing over lustre-MDT0000 [ 5578.453461] Lustre: server umount lustre-MDT0000 complete [ 5582.228321] Lustre: Failing over lustre-MDT0001 [ 5582.724819] Lustre: server umount lustre-MDT0001 complete [ 5592.634539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5592.658822] Lustre: lustre-MDT0000: reset Object Index mappings [ 5592.664366] Lustre: Skipped 1 previous similar message [ 5606.880922] LustreError: 16397:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e2ef971f80 x1874160599568000/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5612.105889] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5623.925025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5624.559629] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:169 to 0x2c0000400:225) [ 5624.560567] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:225) [ 5629.123132] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5630.048773] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:289) [ 5630.051865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:234 to 0x280000401:289) [ 5641.474754] Lustre: DEBUG MARKER: == sanity-scrub test 19: LFSCK can fix multiple linked files on OST ========================================================== 16:27:07 (1787344027) [ 5649.305785] Lustre: server umount lustre-MDT0000 complete [ 5655.521869] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5655.535160] Lustre: Skipped 1 previous similar message [ 5659.373254] Lustre: server umount lustre-MDT0001 complete [ 5663.484468] Lustre: server umount lustre-OST0000 complete [ 5667.690787] Lustre: server umount lustre-OST0001 complete [ 5674.969415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5684.131528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5699.809958] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5704.929711] LustreError: 161995:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 5730.661737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5730.677661] Lustre: Skipped 13 previous similar messages [ 5737.313108] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5742.771295] Lustre: Failing over lustre-OST0000 [ 5742.867675] Lustre: server umount lustre-OST0000 complete [ 5749.955669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5760.109749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5775.714289] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5780.832256] LustreError: 163520:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 5814.331653] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5825.677121] Lustre: DEBUG MARKER: == sanity-scrub test 20: Don't trigger OI scrub for irreparable oi repeatedly ========================================================== 16:30:11 (1787344211) [ 5843.154511] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 5854.253175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.206100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:385) [ 5861.328581] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5871.443703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5871.591036] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5871.599397] Lustre: Skipped 3 previous similar messages [ 5871.805264] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:257) [ 5876.458598] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5880.137552] Lustre: 166431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5880.143017] Lustre: 166431:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 5898.812145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5904.387760] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:169 to 0x2c0000400:257) [ 5904.398066] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:321) [ 5906.366409] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5914.912803] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5921.463915] Lustre: *** cfs_fail_loc=193, val=0*** [ 5923.407335] Lustre: Failing over lustre-MDT0000 [ 5923.638829] Lustre: *** cfs_fail_loc=193, val=0*** [ 5923.875303] Lustre: server umount lustre-MDT0000 complete [ 5932.905315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5933.487239] Lustre: *** cfs_fail_loc=193, val=0*** [ 5938.776298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:417) [ 5938.776687] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:353) [ 5939.419232] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5943.805741] Lustre: *** cfs_fail_loc=193, val=0*** [ 5943.815969] Lustre: Skipped 3 previous similar messages [ 5944.397979] Lustre: *** cfs_fail_loc=19f, val=0*** [ 5944.399654] Lustre: lustre-MDT0000: trigger OI scrub by RPC for [0x200003ab1:0x2:0x0]/16843009 with flags 0x4a: rc = 0 [ 5944.400265] Lustre: Skipped 53 previous similar messages [ 5960.380613] Lustre: DEBUG MARKER: == sanity-scrub test 21: don't hang MDS recovery when failed to get update log ========================================================== 16:32:25 (1787344345) [ 5963.827429] Lustre: Failing over lustre-MDT0000 [ 5964.256599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5964.259258] LustreError: Skipped 7 previous similar messages [ 5966.119705] Lustre: server umount lustre-MDT0000 complete [ 5971.114649] Lustre: Failing over lustre-MDT0001 [ 5971.405937] Lustre: server umount lustre-MDT0001 complete [ 5974.495245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5982.383349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.215721] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6010.825431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.161902] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6017.473905] LustreError: 170935:0:(update_trans.c:1081:top_trans_stop()) lustre-MDT0000-osp-MDT0001: stop trans failed: rc = -20 [ 6017.534470] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:169 to 0x2c0000400:289) [ 6017.535476] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:289) [ 6020.796252] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:385) [ 6020.796252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:449) [ 6023.136495] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 200 [ 6030.786483] Lustre: DEBUG MARKER: == sanity-scrub test 22: LFSCK can recreate or fix the LASTID on MDT/OST ========================================================== 16:33:36 (1787344416) [ 6036.458501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6042.959566] Lustre: server umount lustre-MDT0000 complete [ 6048.662946] Lustre: server umount lustre-MDT0001 complete [ 6062.663777] Lustre: server umount lustre-OST0000 complete [ 6077.861562] Lustre: server umount lustre-OST0001 complete [ 6084.768288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6097.088216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6097.736014] LustreError: 173296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6097.757812] LustreError: 173296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 108 previous similar messages [ 6104.218208] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6110.564846] Lustre: Failing over lustre-MDT0000 [ 6110.570407] LustreError: 173329:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6110.575864] LustreError: 173329:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6110.583659] LustreError: 173329:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 6111.215589] Lustre: server umount lustre-MDT0000 complete [ 6118.623550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6134.093900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 6142.140225] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6153.053486] Lustre: DEBUG MARKER: === sanity-scrub: start setup 16:35:38 (1787344538) === [ 6156.580118] LustreError: 174966:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6156.618666] LustreError: 174966:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6156.642200] LustreError: 174966:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 6157.241763] Lustre: server umount lustre-MDT0000 complete [ 6192.451528] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 6199.445826] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 6252.576487] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 6266.279306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.507899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6266.533408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6266.646754] Lustre: lustre-MDT0000: new disk, initializing [ 6266.840980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6271.279977] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6286.400247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6286.625573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6286.629974] Lustre: Skipped 1 previous similar message [ 6286.747903] Lustre: lustre-MDT0001: new disk, initializing [ 6286.916908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6286.929172] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6292.295368] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6297.488438] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6308.679883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6308.976340] Lustre: lustre-OST0000: new disk, initializing [ 6308.980287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6308.986442] Lustre: 182271:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6311.012833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6311.024116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6311.126133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6316.213984] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6332.231664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6332.428651] Lustre: lustre-OST0001: new disk, initializing [ 6332.432142] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6332.445929] Lustre: 183295:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6332.545888] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6332.556453] Lustre: Skipped 12 previous similar messages [ 6333.680833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6333.701832] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6333.789950] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6339.707782] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6350.171681] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6354.053548] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6360.654731] Lustre: DEBUG MARKER: === sanity-scrub: finish setup 16:39:06 (1787344746) === [ 6362.462763] Lustre: DEBUG MARKER: == sanity-scrub test complete, duration 6073 sec ========= 16:39:08 (1787344748) [ 6364.579744] Lustre: DEBUG MARKER: === sanity-scrub: start cleanup 16:39:09 (1787344749) === [ 6368.132453] Lustre: DEBUG MARKER: === sanity-scrub: finish cleanup 16:39:13 (1787344753) === [ 6373.872869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6373.895281] Lustre: Skipped 28 previous similar messages [ 6373.900321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6373.906332] Lustre: Skipped 4 previous similar messages [ 6378.320182] Lustre: server umount lustre-MDT0000 complete [ 6388.559991] LustreError: 180323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787344775 with bad export cookie 12775008891204226175 [ 6388.561334] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6388.585202] LustreError: 180323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 10 previous similar messages [ 6388.613151] LustreError: Skipped 5 previous similar messages [ 6389.208217] Lustre: server umount lustre-MDT0001 complete [ 6409.707608] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787344780/real 1787344780] req@ffff97e2d0a23b80 x1874160599859968/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787344796 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6409.750768] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 44 previous similar messages [ 6409.842236] Lustre: server umount lustre-OST0000 complete [ 6419.015505] Lustre: server umount lustre-OST0001 complete [ 6439.413625] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 6442.389670] Key type lgssc unregistered [ 6442.844053] LNet: 186708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6442.855730] LNetError: 186708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6442.872983] LNet: Removed LNI 192.168.202.108@tcp [ 6443.816215] Key type .llcrypt unregistered [ 6443.818675] Key type ._llcrypt unregistered