[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 598049635 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001019] APIC: Switch to symmetric I/O mode setup [ 0.002348] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007858] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011198] LSM: Security Framework initializing [ 0.012079] Yama: becoming mindful. [ 0.013061] SELinux: Initializing. [ 0.014079] *** VALIDATE selinux *** [ 0.023798] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028175] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029169] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030189] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032024] *** VALIDATE tmpfs *** [ 0.033487] *** VALIDATE proc *** [ 0.035133] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.038170] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039306] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042031] Spectre V2 : User space: Vulnerable [ 0.043012] Speculative Store Bypass: Vulnerable [ 0.046234] debug: unmapping init [mem 0xffffffff95859000-0xffffffff95860fff] [ 0.049000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049691] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050030] ... version: 2 [ 0.051016] ... bit width: 48 [ 0.052020] ... generic registers: 4 [ 0.053017] ... value mask: 0000ffffffffffff [ 0.054013] ... max period: 00007fffffffffff [ 0.055013] ... fixed-purpose events: 3 [ 0.056010] ... event mask: 000000070000000f [ 0.057306] rcu: Hierarchical SRCU implementation. [ 0.059625] smp: Bringing up secondary CPUs ... [ 0.060709] x86: Booting SMP configuration: [ 0.061042] .... node #0, CPUs: #1 #2 #3 [ 0.067120] smp: Brought up 1 node, 4 CPUs [ 0.069014] smpboot: Max logical packages: 1 [ 0.070037] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.156333] node 0 deferred pages initialised in 84ms [ 0.160257] devtmpfs: initialized [ 0.162496] x86/mm: Memory block size: 128MB [ 0.166756] gcov: version magic: 0x41383552 [ 0.170495] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.174098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.176347] pinctrl core: initialized pinctrl subsystem [ 0.179281] [ 0.179772] ************************************************************* [ 0.181011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184013] ** ** [ 0.186010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.189017] ** ** [ 0.191014] ** This means that this kernel is built to expose internal ** [ 0.193013] ** IOMMU data structures, which may compromise security on ** [ 0.196015] ** your system. ** [ 0.198011] ** ** [ 0.200014] ** If you see this message and you are not debugging the ** [ 0.202014] ** kernel, report this immediately to your vendor! ** [ 0.204017] ** ** [ 0.207015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.209012] ************************************************************* [ 0.212940] NET: Registered protocol family 16 [ 0.214608] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.216203] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.220068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.224050] cpuidle: using governor menu [ 0.225793] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.228605] PCI: Using configuration type 1 for base access [ 0.230154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.238181] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.240020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.243485] cryptd: max_cpu_qlen set to 1000 [ 0.245402] ACPI: Added _OSI(Module Device) [ 0.246000] ACPI: Added _OSI(Processor Device) [ 0.246000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.248013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.253674] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.260647] ACPI: Interpreter enabled [ 0.262153] ACPI: PM: (supports S0 S3 S4 S5) [ 0.263018] ACPI: Using IOAPIC for interrupt routing [ 0.265692] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.268599] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.277695] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.279046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.281020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.284099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.289626] acpiphp: Slot [2] registered [ 0.291155] acpiphp: Slot [5] registered [ 0.292119] acpiphp: Slot [6] registered [ 0.293221] acpiphp: Slot [7] registered [ 0.295124] acpiphp: Slot [8] registered [ 0.296119] acpiphp: Slot [9] registered [ 0.298155] acpiphp: Slot [10] registered [ 0.299119] acpiphp: Slot [3] registered [ 0.301181] acpiphp: Slot [4] registered [ 0.302089] acpiphp: Slot [11] registered [ 0.303092] acpiphp: Slot [12] registered [ 0.304288] acpiphp: Slot [13] registered [ 0.306085] acpiphp: Slot [14] registered [ 0.307083] acpiphp: Slot [15] registered [ 0.309114] acpiphp: Slot [16] registered [ 0.310076] acpiphp: Slot [17] registered [ 0.311161] acpiphp: Slot [18] registered [ 0.313102] acpiphp: Slot [19] registered [ 0.314085] acpiphp: Slot [20] registered [ 0.315089] acpiphp: Slot [21] registered [ 0.316101] acpiphp: Slot [22] registered [ 0.318104] acpiphp: Slot [23] registered [ 0.319000] acpiphp: Slot [24] registered [ 0.320085] acpiphp: Slot [25] registered [ 0.321096] acpiphp: Slot [26] registered [ 0.322165] acpiphp: Slot [27] registered [ 0.324086] acpiphp: Slot [28] registered [ 0.325107] acpiphp: Slot [29] registered [ 0.326094] acpiphp: Slot [30] registered [ 0.327086] acpiphp: Slot [31] registered [ 0.328052] PCI host bridge to bus 0000:00 [ 0.329022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.331022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333145] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.335021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.338032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.340034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.342186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.344010] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.347291] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.357016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.361071] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.372584] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.374820] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.377041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379855] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.384017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.401019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408037] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.419015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.453028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470394] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.480020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.491022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.511015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.522247] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.529014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.539020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.557019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.566000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.574016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.580022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.596024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.604234] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.611015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.633033] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.640963] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.652022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.662020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.686029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.697827] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.700385] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.703792] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.705428] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.708197] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.714144] iommu: Default domain type: Passthrough [ 0.715469] SCSI subsystem initialized [ 0.716174] ACPI: bus type USB registered [ 0.718115] usbcore: registered new interface driver usbfs [ 0.721293] usbcore: registered new interface driver hub [ 0.723241] usbcore: registered new device driver usb [ 0.725166] pps_core: LinuxPPS API ver. 1 registered [ 0.727011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.729059] PTP clock support registered [ 0.731170] EDAC MC: Ver: 3.0.0 [ 0.731890] PCI: Using ACPI for IRQ routing [ 0.732000] NetLabel: Initializing [ 0.732000] NetLabel: domain hash size = 128 [ 0.734013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.736079] NetLabel: unlabeled traffic allowed by default [ 0.738251] vgaarb: loaded [ 0.739452] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.741012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.746166] clocksource: Switched to clocksource kvm-clock [ 0.862162] VFS: Disk quotas dquot_6.6.0 [ 0.863511] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.865486] *** VALIDATE ramfs *** [ 0.866521] *** VALIDATE hugetlbfs *** [ 0.867763] pnp: PnP ACPI init [ 0.870221] pnp: PnP ACPI: found 6 devices [ 0.885678] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.888133] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.889815] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.891493] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.893392] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.895412] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.897596] NET: Registered protocol family 2 [ 0.899660] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.903857] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.906517] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.910598] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.913945] TCP: Hash tables configured (established 65536 bind 65536) [ 0.916322] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.918794] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.921114] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923546] NET: Registered protocol family 1 [ 0.925712] RPC: Registered named UNIX socket transport module. [ 0.927638] RPC: Registered udp transport module. [ 0.928948] RPC: Registered tcp transport module. [ 0.930370] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.932597] NET: Registered protocol family 44 [ 0.933799] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.936530] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.938237] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.939890] PCI: CLS 0 bytes, default 64 [ 0.942158] Unpacking initramfs... [ 2.522200] debug: unmapping init [mem 0xffffa051bcc54000-0xffffa051bffbffff] [ 2.526209] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.528342] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.531232] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.068745] Initialise system trusted keyrings [ 3.076485] Key type blacklist registered [ 3.082975] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.108124] zbud: loaded [ 3.124619] *** VALIDATE nfs *** [ 3.131439] *** VALIDATE nfs4 *** [ 3.133411] pstore: using deflate compression [ 3.145823] Platform Keyring initialized [ 3.812935] NET: Registered protocol family 38 [ 3.814468] Key type asymmetric registered [ 3.815766] Asymmetric key parser 'x509' registered [ 3.824861] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.835824] io scheduler mq-deadline registered [ 3.843542] io scheduler kyber registered [ 3.851640] io scheduler bfq registered [ 3.861042] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.863906] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.874821] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.890244] ACPI: Power Button [PWRF] [ 3.912776] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.953397] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.021684] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.048694] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.096820] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.141275] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.201531] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.211841] Non-volatile memory driver v1.3 [ 4.215725] Linux agpgart interface v0.103 [ 4.335830] virtio_blk virtio1: [vda] 150080 512-byte logical blocks (76.8 MB/73.3 MiB) [ 4.341816] vda: detected capacity change from 0 to 76840960 [ 4.379581] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.386197] vdb: detected capacity change from 0 to 1073741824 [ 4.429771] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.434403] vdc: detected capacity change from 0 to 2621440000 [ 4.495835] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.502386] vdd: detected capacity change from 0 to 2621440000 [ 4.552272] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.556953] vde: detected capacity change from 0 to 4294967296 [ 4.618446] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.627078] vdf: detected capacity change from 0 to 4294967296 [ 4.653105] libphy: Fixed MDIO Bus: probed [ 4.666751] usbcore: registered new interface driver usbserial_generic [ 4.671115] usbserial: USB Serial support registered for generic [ 4.680144] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.686238] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.690579] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.696166] mousedev: PS/2 mouse device common for all mice [ 4.702225] rtc_cmos 00:05: RTC can wake from S4 [ 4.709217] rtc_cmos 00:05: registered as rtc0 [ 4.712721] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.713383] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.721143] intel_pstate: CPU model not supported [ 4.744171] hid: raw HID events driver (C) Jiri Kosina [ 4.750178] usbcore: registered new interface driver usbhid [ 4.758042] usbhid: USB HID core driver [ 4.760695] drop_monitor: Initializing network drop monitor service [ 4.769106] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.776785] Initializing XFRM netlink socket [ 4.780206] NET: Registered protocol family 10 [ 4.784337] Segment Routing with IPv6 [ 4.787427] NET: Registered protocol family 17 [ 4.790716] mpls_gso: MPLS GSO support [ 4.793088] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.801498] RAS: Correctable Errors collector initialized. [ 4.803371] AVX version of gcm_enc/dec engaged. [ 4.805365] AES CTR mode by8 optimization enabled [ 5.045429] sched_clock: Marking stable (5045319784, 0)->(6086070067, -1040750283) [ 5.049522] registered taskstats version 1 [ 5.063799] Loading compiled-in X.509 certificates [ 5.066523] zswap: loaded using pool lzo/zbud [ 5.122786] Key type big_key registered [ 5.143454] Key type encrypted registered [ 5.147627] ima: No TPM chip found, activating TPM-bypass! [ 5.149565] ima: Allocated hash algorithm: sha1 [ 5.151022] ima: No architecture policies found [ 5.155579] evm: Initialising EVM extended attributes: [ 5.166974] evm: security.selinux [ 5.168134] evm: security.ima [ 5.169125] evm: security.capability [ 5.173815] evm: HMAC attrs: 0x1 [ 5.181934] rtc_cmos 00:05: setting system clock to 2026-09-07 03:42:57 UTC (1788752577) [ 5.189953] debug: unmapping init [mem 0xffffffff96803000-0xffffffff969fffff] [ 5.198752] debug: unmapping init [mem 0xffffffff95582000-0xffffffff95858fff] [ 5.232117] Write protecting the kernel read-only data: 28672k [ 5.238105] debug: unmapping init [mem 0xffffffff93c03000-0xffffffff93dfffff] [ 5.247292] debug: unmapping init [mem 0xffffffff94514000-0xffffffff945fffff] [ 5.357512] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.377217] systemd[1]: Detected virtualization kvm. [ 5.380215] systemd[1]: Detected architecture x86-64. [ 5.383109] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.436383] systemd[1]: No hostname configured. [ 5.440578] systemd[1]: Set hostname to . [ 5.445837] random: systemd: uninitialized urandom read (16 bytes read) [ 5.459058] systemd[1]: Initializing machine ID from random generator. [ 5.659966] random: ln: uninitialized urandom read (6 bytes read) [ 5.956904] random: systemd: uninitialized urandom read (16 bytes read) [ 5.968779] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.983706] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.994651] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.928507] device-mapper: uevent: version 1.0.3 [ 7.938682] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.459663] random: fast init done [ 10.470212] virtio_net virtio0 ens2: renamed from eth0 [ 10.767660] scsi host0: ata_piix [ 10.773943] scsi host1: ata_piix [ 10.775179] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.776962] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.703414] random: crng init done [ 15.706203] random: 7 urandom warning(s) missed due to ratelimiting [ 18.435612] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.250142] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.058180] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.146751] SELinux: Disabled at runtime. [ 26.373456] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.384370] systemd[1]: Detected virtualization kvm. [ 26.387320] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.948215] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.959083] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.984874] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.994259] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 29.005230] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.054950] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.075369] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ 29.263735] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 31.117174] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.725044] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.733894] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.596540] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.766920] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 39.733307] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit)[ 40.875242] NFS: Registering the id_resolver key type [ 40.876582] Key type id_resolver registered [ 40.889416] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg131-server login: [ 108.923089] hrtimer: interrupt took 4079371 ns [ 111.477630] libcfs: loading out-of-tree module taints kernel. [ 111.556498] Key type ._llcrypt registered [ 111.558876] Key type .llcrypt registered [ 111.671172] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 133.108942] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 135.009069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 135.040273] alg: No test for adler32 (adler32-zlib) [ 136.531530] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 137.499733] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 139.304878] Key type lgssc registered [ 141.297963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 162.816450] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 208.760560] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 222.975095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 223.016427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.246923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 224.269256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 224.361979] Lustre: lustre-MDT0000: new disk, initializing [ 224.474854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.500785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 229.227607] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.219708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 243.322173] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 243.364435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 243.366809] Lustre: Skipped 1 previous similar message [ 243.444193] Lustre: lustre-MDT0001: new disk, initializing [ 243.524323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.560169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 243.571891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 247.865834] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.723827] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 263.706563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 263.931833] Lustre: lustre-OST0000: new disk, initializing [ 263.936385] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 263.941334] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 263.990604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 269.819439] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.389634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 270.397275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 270.461833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 284.334029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 284.471147] Lustre: lustre-OST0001: new disk, initializing [ 284.480682] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 284.487887] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 284.588280] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 290.568646] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.402879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 291.409544] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 291.500149] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 303.272335] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 309.094397] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 315.256957] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing check_logdir /tmp/testlogs/ [ 319.928659] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing yml_node [ 325.563896] Lustre: DEBUG MARKER: Client: 2.17.58.43 [ 328.510599] Lustre: DEBUG MARKER: MDS: 2.17.58.43 [ 331.088758] Lustre: DEBUG MARKER: OSS: 2.17.58.43 [ 332.890767] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-scrub ============----- Sun Sep 6 23:48:24 EDT 2026 [ 351.211734] Lustre: DEBUG MARKER: excepting tests: [ 361.297419] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 368.610377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.614880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 368.627398] Lustre: Skipped 3 previous similar messages [ 368.636636] Lustre: Skipped 3 previous similar messages [ 373.794421] Lustre: server umount lustre-MDT0000 complete [ 378.864038] LustreError: 6521:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 378.900742] LustreError: 6521:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 382.356275] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752954 with bad export cookie 3736713978989944716 [ 382.356765] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 382.365142] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 382.722227] Lustre: server umount lustre-MDT0001 complete [ 400.352904] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788752956/real 1788752956] req@ffffa0520f695180 x1875643160691456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788752972 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 400.372136] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.703833] Lustre: server umount lustre-OST0000 complete [ 404.449638] Lustre: 3642:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788752960/real 1788752960] req@ffffa0520f695f80 x1875643160691712/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788752976 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 408.609922] Lustre: 3642:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788752964/real 1788752964] req@ffffa051031cbb80 x1875643160692096/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788752980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 410.734468] Lustre: server umount lustre-OST0001 complete [ 425.810022] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 428.126533] Key type lgssc unregistered [ 428.393662] LNet: 14791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.396551] LNetError: 14791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 429.415589] LNet: Removed LNI 192.168.201.131@tcp [ 430.352181] Key type .llcrypt unregistered [ 430.356278] Key type ._llcrypt unregistered [ 454.597983] Key type ._llcrypt registered [ 454.599586] Key type .llcrypt registered [ 454.688275] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 469.151311] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 470.194718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 470.470173] alg: No test for adler32 (adler32-zlib) [ 471.508498] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 471.710964] LNet: Added LNI 192.168.201.131@tcp [8/256/0/180] [ 473.376237] Key type lgssc registered [ 474.351712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 521.225193] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 533.828119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 533.856679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.084673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 535.116336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 535.169560] Lustre: lustre-MDT0000: new disk, initializing [ 535.225910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.242914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 539.425648] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 550.787718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.898354] Lustre: 19241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 550.916454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 550.919594] Lustre: Skipped 1 previous similar message [ 550.970852] Lustre: lustre-MDT0001: new disk, initializing [ 551.022721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 551.043665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 551.052445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 554.500087] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 558.776558] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 568.218562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 568.385262] Lustre: lustre-OST0000: new disk, initializing [ 568.388940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 568.394667] Lustre: 21175:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 568.455569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.264756] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 575.551511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 575.568395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 581.118616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 587.945620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 588.152377] Lustre: lustre-OST0001: new disk, initializing [ 588.157298] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 588.163910] Lustre: 22200:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 588.247734] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 594.461304] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 597.527448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 597.535088] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 597.596643] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 604.362519] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 615.575132] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 630.815968] Lustre: DEBUG MARKER: === sanity-scrub: finish setup 23:53:21 (1788753201) === [ 634.642933] Lustre: DEBUG MARKER: == sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 23:53:24 (1788753204) [ 665.113972] Lustre: Failing over lustre-MDT0000 [ 665.437297] Lustre: server umount lustre-MDT0000 complete [ 668.642863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 668.651763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.032182] LustreError: 19232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753241 with bad export cookie 5880244527068051028 [ 669.033996] Lustre: Failing over lustre-MDT0001 [ 669.034950] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.038375] LustreError: 19232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 669.156782] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.161128] Lustre: Skipped 1 previous similar message [ 669.162368] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 669.285121] Lustre: server umount lustre-MDT0001 complete [ 677.617772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 678.752391] LustreError: 24209:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 678.768121] LustreError: 24209:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffffa05104467b80 x1875643512000768/t0(0) o250->MGC192.168.201.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788753251 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:4294967295 [ 678.799743] LustreError: 24209:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 679.392802] LustreError: 16400:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa051044f3800 x1875643512002816/t0(0) o250->MGC192.168.201.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 679.824799] LustreError: 21169:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.850863] LustreError: 21169:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 679.963546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.002329] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 685.049967] LustreError: 21170:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 685.473791] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 689.215551] LustreError: 24223:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.236929] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 689.290834] LustreError: 24223:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 694.241822] LustreError: 24221:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.266417] LustreError: 24221:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 696.663187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 697.240356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 697.384063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 697.438980] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:65) [ 697.440734] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:65) [ 702.437708] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 702.444453] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 702.459094] Lustre: Skipped 1 previous similar message [ 702.510144] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 702.599206] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:65) [ 702.599408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 704.506280] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 720.406348] Lustre: DEBUG MARKER: == sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 23:54:52 (1788753292) [ 744.168846] Lustre: Failing over lustre-MDT0000 [ 744.461825] Lustre: server umount lustre-MDT0000 complete [ 748.513864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 748.514282] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.521247] LustreError: 25158:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 748.542531] Lustre: Skipped 3 previous similar messages [ 748.569126] LustreError: 25158:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 748.588380] Lustre: Failing over lustre-MDT0001 [ 748.589040] LustreError: 19233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753320 with bad export cookie 5880244527068067352 [ 748.589435] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 748.614412] LustreError: 19233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 748.958567] Lustre: server umount lustre-MDT0001 complete [ 758.455333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 769.001464] Lustre: 16402:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788753325/real 1788753325] req@ffffa051038b4700 x1875643512122496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788753341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 769.048563] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.078076] Lustre: Skipped 1 previous similar message [ 773.217342] LustreError: 16400:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa051038b7b80 x1875643512124544/t0(0) o250->MGC192.168.201.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 773.520382] LustreError: 21170:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.624771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 773.676753] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 774.624593] Lustre: 16404:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788753330/real 1788753330] req@ffffa051038b4a80 x1875643512123136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788753346 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 774.677969] Lustre: 16404:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 775.654341] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788753331/real 1788753331] req@ffffa0510acb9f80 x1875643512123392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788753347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 775.678486] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 778.372195] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 778.721113] Lustre: 16403:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788753335/real 1788753335] req@ffffa05102000000 x1875643512123520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788753351 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 778.757056] Lustre: 16403:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 783.840733] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788753340/real 1788753340] req@ffffa051038b5500 x1875643512124160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788753356 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 783.893959] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 787.931127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 787.956150] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 787.980377] Lustre: Skipped 2 previous similar messages [ 788.226059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 788.381404] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:106 to 0x280000400:129) [ 788.381953] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:105 to 0x2c0000400:129) [ 792.970728] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 793.571519] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 793.575429] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 793.592933] Lustre: Skipped 1 previous similar message [ 793.628592] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 793.682866] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 793.687320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:129) [ 796.946662] Lustre: *** cfs_fail_loc=193, val=0*** [ 802.212342] Lustre: Failing over lustre-MDT0000 [ 802.803040] Lustre: server umount lustre-MDT0000 complete [ 803.823213] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 803.824996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 803.839465] Lustre: Skipped 3 previous similar messages [ 803.849705] LustreError: 27876:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 803.899974] LustreError: 27876:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 13 previous similar messages [ 811.782372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 812.002101] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 812.312210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 812.319609] Lustre: Skipped 1 previous similar message [ 812.380037] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 817.486344] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 817.640101] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 817.648430] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 817.669978] Lustre: Skipped 2 previous similar messages [ 817.703993] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 817.748281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:161) [ 817.749121] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:161) [ 828.534591] Lustre: DEBUG MARKER: == sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 23:56:39 (1788753399) [ 846.803876] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 867.496767] Lustre: 30498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 890.784569] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 894.537436] Lustre: 31634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 908.170749] Lustre: *** cfs_fail_loc=198, val=0*** [ 919.382141] Lustre: Failing over lustre-MDT0000 [ 919.751990] Lustre: server umount lustre-MDT0000 complete [ 920.034057] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.037124] LustreError: 21170:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.063058] Lustre: Skipped 4 previous similar messages [ 920.107582] LustreError: 21170:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 924.330931] LustreError: 25743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753496 with bad export cookie 5880244527068096374 [ 924.332266] Lustre: Failing over lustre-MDT0001 [ 924.332718] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 924.341296] LustreError: 25743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 924.856684] Lustre: server umount lustre-MDT0001 complete [ 930.768385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 937.374722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 941.472924] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788753497/real 1788753497] req@ffffa0522f6cd880 x1875643512297216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788753513 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 941.502894] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 941.509209] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.300360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 948.707345] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x519ad33e145491f0 [ 948.718852] Lustre: MGC192.168.201.131@tcp: Connection restored to 0@lo (at 0@lo) [ 948.736060] Lustre: Skipped 3 previous similar messages [ 949.278907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 949.322056] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 953.839648] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 962.309506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 962.500896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 962.641384] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:193) [ 962.641632] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:193) [ 967.507846] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 967.653337] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 967.657438] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 967.671923] Lustre: Skipped 2 previous similar messages [ 967.705953] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 967.753953] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:202 to 0x2c0000401:225) [ 967.754108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:202 to 0x280000401:225) [ 979.864562] Lustre: DEBUG MARKER: == sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 23:59:11 (1788753551) [ 1001.137510] Lustre: Failing over lustre-MDT0000 [ 1003.431370] Lustre: server umount lustre-MDT0000 complete [ 1003.491410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1003.500435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1003.512519] Lustre: Skipped 1 previous similar message [ 1003.516338] LustreError: 33112:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1003.543253] LustreError: 33112:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 10 previous similar messages [ 1007.014915] LustreError: 19231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753579 with bad export cookie 5880244527068123632 [ 1007.016741] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1007.016991] Lustre: Failing over lustre-MDT0001 [ 1007.028122] LustreError: 19231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1007.347860] Lustre: server umount lustre-MDT0001 complete [ 1012.578994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1016.915316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1023.904969] Lustre: 16403:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788753580/real 1788753580] req@ffffa051044f2300 x1875643512416128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788753596 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1023.923660] Lustre: 16403:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 1025.397095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1025.468986] Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64 [ 1032.488966] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1037.669263] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1042.918913] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1042.943126] Lustre: Skipped 2 previous similar messages [ 1048.617463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1048.699894] Lustre: lustre-MDT0001: invalid oi count 63, remove them, then set it to 64 [ 1049.047696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1049.304107] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:234 to 0x2c0000400:257) [ 1049.309301] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:233 to 0x280000400:257) [ 1054.557875] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1054.698497] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1054.737335] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1054.806150] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:266 to 0x2c0000401:289) [ 1054.807939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 1077.855921] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1097.485295] Lustre: 38944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1121.730366] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1125.548365] Lustre: 40081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1149.798949] Lustre: Failing over lustre-MDT0000 [ 1150.174365] Lustre: server umount lustre-MDT0000 complete [ 1151.971744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1151.977486] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.992744] LustreError: 21169:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.992756] LustreError: 21169:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 16 previous similar messages [ 1152.034225] Lustre: Skipped 8 previous similar messages [ 1154.689875] LustreError: 19233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753727 with bad export cookie 5880244527068151205 [ 1154.691830] Lustre: Failing over lustre-MDT0001 [ 1154.694478] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1154.698127] LustreError: 19233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1154.994799] Lustre: server umount lustre-MDT0001 complete [ 1160.491215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1170.121376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1173.474107] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788753729/real 1788753729] req@ffffa0520fe56d80 x1875643512563072/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788753745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1173.519066] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 1182.904255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1182.981726] Lustre: lustre-MDT0000: invalid oi count 58, remove them, then set it to 64 [ 1200.225693] LustreError: 16400:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0521d5f3b80 x1875643512566016/t0(0) o250->MGC192.168.201.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1200.709195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.718663] Lustre: Skipped 3 previous similar messages [ 1200.764688] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1205.858501] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1215.599245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1215.658235] Lustre: lustre-MDT0001: invalid oi count 58, remove them, then set it to 64 [ 1216.071087] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1216.136931] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:298 to 0x2c0000400:321) [ 1216.141074] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:297 to 0x280000400:321) [ 1217.143183] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1217.155309] Lustre: Skipped 4 previous similar messages [ 1217.189635] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1217.273596] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:329 to 0x2c0000401:353) [ 1217.274289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:330 to 0x280000401:353) [ 1221.076180] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1243.312579] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1261.323718] Lustre: 44791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1284.865284] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1288.256380] Lustre: 45928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1309.335151] Lustre: Failing over lustre-MDT0000 [ 1309.642747] Lustre: server umount lustre-MDT0000 complete [ 1309.664878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1309.670530] LustreError: Skipped 1 previous similar message [ 1309.672480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.677494] Lustre: Skipped 2 previous similar messages [ 1313.644286] LustreError: 19232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753885 with bad export cookie 5880244527068178575 [ 1313.646405] Lustre: Failing over lustre-MDT0001 [ 1313.648532] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1313.656076] LustreError: 19232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1314.182437] Lustre: server umount lustre-MDT0001 complete [ 1319.040095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1326.286237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1332.199041] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788753888/real 1788753888] req@ffffa0510327c700 x1875643512712576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788753904 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1332.231652] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 12 previous similar messages [ 1338.683205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1338.730569] Lustre: lustre-MDT0000: invalid oi count 56, remove them, then set it to 64 [ 1358.350694] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1364.532626] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1374.078213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1374.111159] Lustre: lustre-MDT0001: invalid oi count 56, remove them, then set it to 64 [ 1374.650716] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:361 to 0x2c0000400:385) [ 1374.658282] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:362 to 0x280000400:385) [ 1376.610288] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1376.614859] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1376.648838] Lustre: Skipped 4 previous similar messages [ 1376.697795] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1376.791845] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:394 to 0x2c0000401:417) [ 1376.793938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:393 to 0x280000401:417) [ 1383.058712] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1399.710439] Lustre: DEBUG MARKER: == sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 00:06:11 (1788753971) [ 1413.836704] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 1433.932315] Lustre: 50639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1458.327651] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1489.025237] Lustre: Failing over lustre-MDT0000 [ 1489.377818] LustreError: 25158:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.393288] LustreError: 25158:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 16 previous similar messages [ 1491.349686] Lustre: server umount lustre-MDT0000 complete [ 1495.022936] Lustre: Failing over lustre-MDT0001 [ 1495.024892] LustreError: 19231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754067 with bad export cookie 5880244527068205945 [ 1495.039278] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1495.071185] LustreError: 19231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1495.366852] Lustre: server umount lustre-MDT0001 complete [ 1501.904396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1510.880428] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788754067/real 1788754067] req@ffffa0521b63c000 x1875643512870400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788754083 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1510.897340] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 1516.269446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1526.627105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1537.957772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1552.709427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1552.723571] Lustre: lustre-MDT0000: reset Object Index mappings [ 1565.153145] LustreError: 16400:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0520730f100 x1875643512873472/t0(0) o250->MGC192.168.201.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1565.809353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1565.812230] Lustre: Skipped 3 previous similar messages [ 1565.884751] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1571.367683] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1581.572042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1581.594212] Lustre: lustre-MDT0001: reset Object Index mappings [ 1581.960853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1581.965040] LustreError: Skipped 2 previous similar messages [ 1582.043640] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1582.263559] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:426 to 0x280000400:449) [ 1582.275234] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:425 to 0x2c0000400:449) [ 1587.663333] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1587.740946] Lustre: lustre-MDT0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 1587.813031] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:457 to 0x2c0000401:481) [ 1587.815062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:458 to 0x280000401:481) [ 1600.541804] Lustre: DEBUG MARKER: == sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 00:09:32 (1788754172) [ 1623.381907] Lustre: Failing over lustre-MDT0000 [ 1623.521443] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1623.524429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1623.532867] Lustre: Skipped 14 previous similar messages [ 1623.540701] Lustre: Skipped 4 previous similar messages [ 1623.824198] Lustre: server umount lustre-MDT0000 complete [ 1628.183987] LustreError: 19231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754200 with bad export cookie 5880244527068233315 [ 1628.186814] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1628.194508] LustreError: 19231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1628.200124] Lustre: Failing over lustre-MDT0001 [ 1628.682963] Lustre: server umount lustre-MDT0001 complete [ 1634.983538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1646.097407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1654.485317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1665.046460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1679.221618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1679.242903] Lustre: lustre-MDT0000: reset Object Index mappings [ 1698.144960] LustreError: 16400:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0520b7c0700 x1875643512990336/t0(0) o250->MGC192.168.201.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1698.647461] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1703.733676] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1712.245401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1712.253996] Lustre: lustre-MDT0001: reset Object Index mappings [ 1712.773177] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:490 to 0x280000400:513) [ 1712.782870] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:489 to 0x2c0000400:513) [ 1713.781291] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1713.795922] Lustre: Skipped 9 previous similar messages [ 1713.807213] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1713.842262] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1713.895467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:521 to 0x280000401:545) [ 1713.896720] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:522 to 0x2c0000401:545) [ 1718.156430] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1726.207747] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004282:0x1:0x0]/32003: rc = 0 [ 1727.429177] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240003ab1:0x1:0x0]/32003 with flags 0x52: rc = 0 [ 1749.831366] Lustre: DEBUG MARKER: == sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 00:12:01 (1788754321) [ 1774.044209] Lustre: Failing over lustre-MDT0000 [ 1774.413889] Lustre: server umount lustre-MDT0000 complete [ 1779.407883] LustreError: 19231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754351 with bad export cookie 5880244527068260615 [ 1779.412500] Lustre: Failing over lustre-MDT0001 [ 1779.423951] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1779.441142] LustreError: 19231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1779.933099] Lustre: server umount lustre-MDT0001 complete [ 1786.659380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1796.897249] Lustre: 16404:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788754353/real 1788754353] req@ffffa0521f33f100 x1875643513109760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788754369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1796.946950] Lustre: 16404:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 1798.396910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1807.080389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1818.006584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1831.511376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1831.526974] Lustre: lustre-MDT0000: reset Object Index mappings [ 1849.945766] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1854.993473] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1865.336142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1865.357862] Lustre: lustre-MDT0001: reset Object Index mappings [ 1865.881157] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:554 to 0x2c0000400:577) [ 1865.895222] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:553 to 0x280000400:577) [ 1866.853570] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1866.919736] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1866.981312] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:585 to 0x2c0000401:609) [ 1866.990262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:586 to 0x280000401:609) [ 1871.596357] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1881.232260] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004a52:0x1:0x0]/32045: rc = 0 [ 1884.473905] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004281:0x1:0x0]/32032 with flags 0x52: rc = 0 [ 2006.119791] Lustre: DEBUG MARKER: == sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ========================================================== 00:16:18 (1788754578) [ 2041.488690] Lustre: Failing over lustre-MDT0000 [ 2041.721221] Lustre: server umount lustre-MDT0000 complete [ 2042.339787] LustreError: 63648:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.356561] LustreError: 63648:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 25 previous similar messages [ 2045.627167] Lustre: Failing over lustre-MDT0001 [ 2045.628388] LustreError: 19231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754617 with bad export cookie 5880244527068288006 [ 2045.630571] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2045.661359] LustreError: 19231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2045.983669] Lustre: server umount lustre-MDT0001 complete [ 2051.510584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2062.303599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2071.154037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2081.130551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2093.351763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2093.388473] Lustre: lustre-MDT0000: reset Object Index mappings [ 2115.040456] LustreError: 16400:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0520b742d80 x1875643513310464/t0(0) o250->MGC192.168.201.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2115.437095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.440193] Lustre: Skipped 5 previous similar messages [ 2115.487977] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2120.569389] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2129.934635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2129.984309] Lustre: lustre-MDT0001: reset Object Index mappings [ 2130.260864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2130.263642] LustreError: Skipped 3 previous similar messages [ 2130.399047] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:618 to 0x2c0000400:641) [ 2130.407225] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:617 to 0x280000400:641) [ 2132.386557] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2132.418286] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2132.462626] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:649 to 0x2c0000401:673) [ 2132.463106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:650 to 0x280000401:673) [ 2134.663811] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2144.158708] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200005222:0x1:0x0]/64037: rc = 0 [ 2147.378348] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004a51:0x1:0x0]/32003 with flags 0x52: rc = 0 [ 2235.640508] Lustre: DEBUG MARKER: == sanity-scrub test 4d: FID in LMA mismatch with object FID won't block create ========================================================== 00:20:07 (1788754807) [ 2251.241144] Lustre: *** cfs_fail_loc=19b, val=0*** [ 2251.247379] Lustre: Skipped 1 previous similar message [ 2251.742838] Lustre: *** cfs_fail_loc=19b, val=0*** [ 2251.745969] Lustre: Skipped 33 previous similar messages [ 2252.754634] Lustre: *** cfs_fail_loc=19b, val=0*** [ 2252.756225] Lustre: Skipped 235 previous similar messages [ 2271.654238] Lustre: DEBUG MARKER: == sanity-scrub test 4e: FID reuse can be fixed ========== 00:20:43 (1788754843) [ 2276.452139] Lustre: *** cfs_fail_loc=1a0, val=0*** [ 2276.454019] Lustre: Skipped 185 previous similar messages [ 2276.750497] LustreError: 31654:0:(osd_compat.c:737:osd_obj_update_entry()) lustre-OST0000: the FID [0x280000401:0x321:0x0] is used by two objects: 264/3479081981 265/3183458233 [ 2291.487502] Lustre: DEBUG MARKER: == sanity-scrub test 5: OI scrub state machine =========== 00:21:02 (1788754862) [ 2316.777208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.791751] Lustre: Skipped 14 previous similar messages [ 2316.796918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2317.800273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2317.808679] Lustre: Skipped 2 previous similar messages [ 2322.648899] Lustre: server umount lustre-MDT0000 complete [ 2326.870508] LustreError: 19233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754899 with bad export cookie 5880244527068332239 [ 2326.877595] LustreError: 19233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2328.033216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2328.040521] Lustre: Skipped 1 previous similar message [ 2331.046378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2331.049255] Lustre: Skipped 1 previous similar message [ 2336.225176] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2336.235635] Lustre: Skipped 1 previous similar message [ 2341.345397] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2341.575908] Lustre: server umount lustre-MDT0001 complete [ 2359.267039] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2359.345909] Lustre: server umount lustre-OST0000 complete [ 2369.681351] Lustre: server umount lustre-OST0001 complete [ 2375.234738] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 2381.540619] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2421.691163] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 2431.479905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.663598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2431.686319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2431.768648] Lustre: lustre-MDT0000: new disk, initializing [ 2431.884074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2436.535571] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2447.535233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.684407] Lustre: 75848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2447.701493] Lustre: 75848:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2447.729128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2447.732498] Lustre: Skipped 1 previous similar message [ 2447.793765] Lustre: lustre-MDT0001: new disk, initializing [ 2447.890392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2447.906615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2452.315230] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2457.014689] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2464.365991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2464.705215] Lustre: lustre-OST0000: new disk, initializing [ 2464.712845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2464.722392] Lustre: 77483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2466.039029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2466.056017] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2466.226467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2472.148457] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2483.176223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2483.385972] Lustre: lustre-OST0001: new disk, initializing [ 2483.393848] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2483.402906] Lustre: 78356:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2485.323606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2485.342190] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2485.387190] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2490.809291] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2500.615282] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2504.076121] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2526.952053] Lustre: Failing over lustre-MDT0000 [ 2529.257855] Lustre: server umount lustre-MDT0000 complete [ 2533.530385] Lustre: Failing over lustre-MDT0001 [ 2534.009628] Lustre: server umount lustre-MDT0001 complete [ 2540.056095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2550.349914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2553.325279] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788755109/real 1788755109] req@ffffa0520eab7800 x1875643513640320/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788755125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2553.348053] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [ 2558.682369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2567.723181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2581.305536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2581.331428] Lustre: lustre-MDT0000: reset Object Index mappings [ 2608.904889] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2617.888511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2618.271867] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:42 to 0x2c0000400:65) [ 2618.275306] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:65) [ 2622.317355] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2622.323624] Lustre: Skipped 14 previous similar messages [ 2622.401742] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:65) [ 2622.402034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 2623.160331] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2633.785872] Lustre: *** cfs_fail_loc=190, val=3*** [ 2633.786718] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000402:0x1:0x0]/32005: rc = 0 [ 2634.882375] Lustre: *** cfs_fail_loc=190, val=3*** [ 2634.887440] Lustre: Skipped 1 previous similar message [ 2635.963077] Lustre: *** cfs_fail_loc=190, val=3*** [ 2637.142287] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240000402:0x1:0x0]/64025 with flags 0x52: rc = 0 [ 2639.008527] Lustre: *** cfs_fail_loc=190, val=3*** [ 2639.015990] Lustre: Skipped 2 previous similar messages [ 2643.233137] Lustre: *** cfs_fail_loc=190, val=3*** [ 2643.235124] Lustre: Skipped 2 previous similar messages [ 2649.441575] Lustre: Failing over lustre-MDT0000 [ 2649.958989] Lustre: server umount lustre-MDT0000 complete [ 2653.152970] LustreError: 77477:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.170111] LustreError: 77477:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 23 previous similar messages [ 2653.896583] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2653.903577] LustreError: Skipped 2 previous similar messages [ 2653.909274] Lustre: Failing over lustre-MDT0001 [ 2654.200640] Lustre: server umount lustre-MDT0001 complete [ 2664.433358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2678.753022] LustreError: 16400:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0522fc05f80 x1875643513682048/t0(0) o250->MGC192.168.201.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2679.168306] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2679.176494] Lustre: Skipped 1 previous similar message [ 2684.400412] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2693.217798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2693.550620] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:42 to 0x2c0000400:97) [ 2693.558255] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:97) [ 2697.861600] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2698.742316] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2698.749411] Lustre: Skipped 1 previous similar message [ 2698.805119] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2698.812536] Lustre: Skipped 1 previous similar message [ 2698.861331] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:97) [ 2698.871187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:97) [ 2703.127881] Lustre: Failing over lustre-MDT0000 [ 2703.480488] Lustre: server umount lustre-MDT0000 complete [ 2708.089805] Lustre: Failing over lustre-MDT0001 [ 2708.405290] Lustre: server umount lustre-MDT0001 complete [ 2719.086788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2719.146795] Lustre: *** cfs_fail_loc=190, val=3*** [ 2719.150497] Lustre: Skipped 2 previous similar messages [ 2733.539847] LustreError: 16400:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0520fec7100 x1875643513709184/t0(0) o250->MGC192.168.201.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2733.957501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.967146] Lustre: Skipped 9 previous similar messages [ 2737.312303] Lustre: *** cfs_fail_loc=190, val=3*** [ 2737.316895] Lustre: Skipped 5 previous similar messages [ 2738.630882] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2747.683690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2748.016516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2748.036105] LustreError: Skipped 6 previous similar messages [ 2748.194423] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:129) [ 2748.225940] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:42 to 0x2c0000400:129) [ 2752.695172] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2753.607706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:129) [ 2753.610937] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:129) [ 2764.989200] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240000402:0x1:0x0]/64025 with flags 0x52: rc = 0 [ 2765.001459] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000402:0x45:0x0]/152: rc = 0 [ 2776.104277] Lustre: DEBUG MARKER: == sanity-scrub test 6: OI scrub resumes from last checkpoint ========================================================== 00:29:07 (1788755347) [ 2801.234144] Lustre: Failing over lustre-MDT0000 [ 2801.450732] Lustre: server umount lustre-MDT0000 complete [ 2804.604543] Lustre: Failing over lustre-MDT0001 [ 2804.708421] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2804.710925] Lustre: Skipped 3 previous similar messages [ 2804.820753] Lustre: server umount lustre-MDT0001 complete [ 2809.691782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2819.704255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2828.256396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2838.937310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2851.435802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2851.459420] Lustre: lustre-MDT0000: reset Object Index mappings [ 2851.462703] Lustre: Skipped 1 previous similar message [ 2874.339669] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x519ad33e145a2aa4 [ 2878.164942] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2888.332573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2888.904908] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:193) [ 2888.935568] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2893.827428] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2894.406214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 2894.406777] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:170 to 0x2c0000401:193) [ 2903.958644] Lustre: *** cfs_fail_loc=190, val=2*** [ 2903.959208] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200001b72:0x1:0x0]/66: rc = 0 [ 2903.960741] Lustre: Skipped 12 previous similar messages [ 2907.124338] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240001b71:0x1:0x0]/32003 with flags 0x52: rc = 0 [ 2933.170962] Lustre: Failing over lustre-MDT0000 [ 2933.881333] Lustre: server umount lustre-MDT0000 complete [ 2935.274118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2935.289850] Lustre: Skipped 26 previous similar messages [ 2938.101188] LustreError: 78355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755510 with bad export cookie 5880244527068490404 [ 2938.101317] Lustre: Failing over lustre-MDT0001 [ 2938.109760] LustreError: 78355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2938.310645] Lustre: server umount lustre-MDT0001 complete [ 2947.943083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2948.578633] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x519ad33e145a3214 [ 2951.231497] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2956.684373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2956.835573] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:225) [ 2956.836334] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:225) [ 2960.126589] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2961.987684] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:170 to 0x2c0000401:225) [ 2961.989780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:225) [ 2981.887956] Lustre: DEBUG MARKER: == sanity-scrub test 7: System is available during OI scrub scanning ========================================================== 00:32:33 (1788755553) [ 3004.230967] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3029.409163] Lustre: 96539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3054.025680] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3058.974821] Lustre: 97676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3106.066762] Lustre: Failing over lustre-MDT0000 [ 3106.837708] Lustre: server umount lustre-MDT0000 complete [ 3111.976717] Lustre: Failing over lustre-MDT0001 [ 3112.626097] Lustre: server umount lustre-MDT0001 complete [ 3119.616412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3131.056718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3140.747622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3152.112220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3164.558779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3164.588730] Lustre: lustre-MDT0000: reset Object Index mappings [ 3164.592229] Lustre: Skipped 1 previous similar message [ 3182.050812] LustreError: 16400:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0520e675880 x1875643514051328/t0(0) o250->MGC192.168.201.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3188.216619] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3196.500156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3196.935689] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:265 to 0x280000400:289) [ 3196.943413] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:266 to 0x2c0000400:289) [ 3200.080523] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:265 to 0x2c0000401:289) [ 3200.088933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:266 to 0x280000401:289) [ 3202.558815] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3210.906287] Lustre: *** cfs_fail_loc=190, val=3*** [ 3210.907363] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200002b12:0x1:0x0]/64001: rc = 0 [ 3210.908788] Lustre: Skipped 34 previous similar messages [ 3210.928571] Lustre: Skipped 2 previous similar messages [ 3214.153801] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240002b11:0x1:0x0]/32019 with flags 0x52: rc = 0 [ 3233.523840] Lustre: DEBUG MARKER: == sanity-scrub test 8: Control OI scrub manually ======== 00:36:45 (1788755805) [ 3278.628170] Lustre: Failing over lustre-MDT0000 [ 3278.861450] Lustre: server umount lustre-MDT0000 complete [ 3281.891352] LustreError: 77477:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.900871] LustreError: 77477:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 49 previous similar messages [ 3282.586315] Lustre: Failing over lustre-MDT0001 [ 3282.587628] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3282.596616] LustreError: Skipped 4 previous similar messages [ 3282.874754] Lustre: server umount lustre-MDT0001 complete [ 3288.623497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3298.307667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3299.296451] Lustre: 16404:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788755855/real 1788755855] req@ffffa051096c9180 x1875643514184832/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788755871 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3299.330206] Lustre: 16404:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 64 previous similar messages [ 3308.884647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3320.291867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3332.605324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3352.544619] LustreError: 16400:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0520fec6d80 x1875643514189568/t0(0) o250->MGC192.168.201.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3353.429834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3353.448792] Lustre: Skipped 7 previous similar messages [ 3353.527836] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3353.530661] Lustre: Skipped 4 previous similar messages [ 3359.640965] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3368.940224] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3368.945221] Lustre: Skipped 31 previous similar messages [ 3370.723584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3370.960837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 3370.975276] LustreError: Skipped 5 previous similar messages [ 3371.019363] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3371.032321] Lustre: Skipped 4 previous similar messages [ 3371.130309] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:329 to 0x2c0000400:353) [ 3371.133160] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:330 to 0x280000400:353) [ 3375.897125] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3376.205546] Lustre: lustre-MDT0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 3376.222551] Lustre: Skipped 4 previous similar messages [ 3376.296754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:330 to 0x280000401:353) [ 3376.311676] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:329 to 0x2c0000401:353) [ 3384.341889] Lustre: *** cfs_fail_loc=190, val=1*** [ 3384.343567] Lustre: Skipped 10 previous similar messages [ 3416.219368] Lustre: DEBUG MARKER: == sanity-scrub test 9: OI scrub speed control =========== 00:39:47 (1788755987) [ 3433.454317] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3453.582883] Lustre: 109215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3479.443752] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3605.872460] Lustre: Failing over lustre-MDT0000 [ 3606.497360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.510651] Lustre: Skipped 16 previous similar messages [ 3606.591961] Lustre: server umount lustre-MDT0000 complete [ 3611.500952] LustreError: 77111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756183 with bad export cookie 5880244527068632686 [ 3611.513104] Lustre: Failing over lustre-MDT0001 [ 3611.515734] LustreError: 77111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3611.619587] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3611.623514] Lustre: Skipped 1 previous similar message [ 3612.254624] Lustre: server umount lustre-MDT0001 complete [ 3618.020406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3632.693876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3646.044071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3662.306340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3680.204407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3680.234812] Lustre: lustre-MDT0000: reset Object Index mappings [ 3680.237236] Lustre: Skipped 3 previous similar messages [ 3681.318658] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x519ad33e145f8abe [ 3687.881856] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3698.624500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3699.173543] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:394 to 0x2c0000400:417) [ 3699.173926] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:393 to 0x280000400:417) [ 3702.294474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:393 to 0x280000401:417) [ 3702.300244] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:394 to 0x2c0000401:417) [ 3704.604705] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3767.688588] Lustre: DEBUG MARKER: == sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount (1) ========================================================== 00:45:39 (1788756339) [ 3784.501457] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 3809.136434] Lustre: 117189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3809.159676] Lustre: 117189:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 3835.325798] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3992.560869] Lustre: Failing over lustre-MDT0000 [ 3992.810880] Lustre: server umount lustre-MDT0000 complete [ 3994.085990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3994.102348] LustreError: Skipped 2 previous similar messages [ 3994.138901] LustreError: 77478:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.165696] LustreError: 77478:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 17 previous similar messages [ 3996.945308] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3996.954173] LustreError: Skipped 1 previous similar message [ 3996.961687] Lustre: Failing over lustre-MDT0001 [ 3997.349389] Lustre: server umount lustre-MDT0001 complete [ 4002.776813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4014.011714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4017.632196] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788756573/real 1788756573] req@ffffa05210379c00 x1875643514665344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788756589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4017.650306] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 16 previous similar messages [ 4022.697822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4031.820451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4045.208799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4066.848357] LustreError: 16400:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0523a728a80 x1875643514669056/t0(0) o250->MGC192.168.201.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4067.398847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4067.404645] Lustre: Skipped 3 previous similar messages [ 4067.455951] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4067.470028] Lustre: Skipped 1 previous similar message [ 4072.674948] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4081.637591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4081.986407] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:458 to 0x280000400:481) [ 4081.993377] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:457 to 0x2c0000400:481) [ 4086.062411] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4086.068486] Lustre: Skipped 10 previous similar messages [ 4086.070443] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4086.075769] Lustre: Skipped 1 previous similar message [ 4086.104375] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4086.109769] Lustre: Skipped 1 previous similar message [ 4086.154823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:457 to 0x280000401:481) [ 4086.155061] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:458 to 0x2c0000401:481) [ 4086.427766] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4095.516249] Lustre: *** cfs_fail_loc=190, val=1*** [ 4095.517909] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004282:0x1:0x0]/32032: rc = 0 [ 4095.519218] Lustre: Skipped 29 previous similar messages [ 4098.844356] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004281:0x1:0x0]/32003 with flags 0x52: rc = 0 [ 4107.904280] Lustre: Failing over lustre-MDT0000 [ 4108.809279] Lustre: server umount lustre-MDT0000 complete [ 4113.318459] Lustre: Failing over lustre-MDT0001 [ 4113.718357] Lustre: server umount lustre-MDT0001 complete [ 4124.806528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4138.274324] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x519ad33e146a4304 [ 4144.454522] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4154.259294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4154.823781] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:457 to 0x2c0000400:513) [ 4154.827152] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:458 to 0x280000400:513) [ 4159.348485] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4160.072824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:457 to 0x280000401:513) [ 4160.081189] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:458 to 0x2c0000401:513) [ 4165.225261] Lustre: Failing over lustre-MDT0000 [ 4165.576967] Lustre: server umount lustre-MDT0000 complete [ 4169.811142] Lustre: Failing over lustre-MDT0001 [ 4170.447220] Lustre: server umount lustre-MDT0001 complete [ 4181.896245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4194.723592] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x519ad33e146a4963 [ 4200.380694] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4208.999209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4209.525709] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:458 to 0x280000400:545) [ 4209.537267] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:457 to 0x2c0000400:545) [ 4210.651388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:457 to 0x280000401:545) [ 4210.659311] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:458 to 0x2c0000401:545) [ 4215.597889] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4231.626454] Lustre: DEBUG MARKER: == sanity-scrub test 11: OI scrub skips the new created objects only once ========================================================== 00:53:23 (1788756803) [ 4245.110931] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4264.684992] Lustre: 128209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4264.697111] Lustre: 128209:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 4287.260802] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4316.392333] Lustre: DEBUG MARKER: == sanity-scrub test 12: OI scrub can rebuild invalid /O entries ========================================================== 00:54:48 (1788756888) [ 4321.859540] Lustre: *** cfs_fail_loc=195, val=0*** [ 4326.493523] Lustre: Failing over lustre-OST0000 [ 4326.661429] Lustre: server umount lustre-OST0000 complete [ 4327.398276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4327.408263] Lustre: Skipped 21 previous similar messages [ 4335.786365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4343.012196] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4536.755649] Lustre: DEBUG MARKER: == sanity-scrub test 13: OI scrub can rebuild missed /O entries ========================================================== 00:58:28 (1788757108) [ 4541.733772] Lustre: *** cfs_fail_loc=196, val=0*** [ 4541.738712] Lustre: Skipped 63 previous similar messages [ 4546.951232] Lustre: Failing over lustre-OST0000 [ 4547.078743] Lustre: server umount lustre-OST0000 complete [ 4555.697090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4562.380097] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4754.809789] Lustre: DEBUG MARKER: == sanity-scrub test 14: OI scrub can repair OST objects under lost+found ========================================================== 01:02:06 (1788757326) [ 4762.642123] Lustre: *** cfs_fail_loc=196, val=0*** [ 4762.647970] Lustre: Skipped 63 previous similar messages [ 4765.134116] Lustre: *** cfs_fail_loc=196, val=0*** [ 4765.137401] Lustre: Skipped 191 previous similar messages [ 4769.220475] Lustre: *** cfs_fail_loc=196, val=0*** [ 4769.227363] Lustre: Skipped 287 previous similar messages [ 4784.252156] Lustre: Failing over lustre-OST0000 [ 4784.426294] Lustre: server umount lustre-OST0000 complete [ 4786.668633] LustreError: 77466:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.690770] LustreError: 77466:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 39 previous similar messages [ 4796.859491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4797.343950] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4797.364667] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4797.377735] Lustre: Skipped 4 previous similar messages [ 4799.338086] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4799.343628] Lustre: Skipped 4 previous similar messages [ 4799.356781] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4799.356920] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4799.368989] Lustre: Skipped 4 previous similar messages [ 4799.385018] Lustre: Skipped 20 previous similar messages [ 4805.398602] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4817.911283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4817.921331] Lustre: Skipped 2 previous similar messages [ 4819.231951] Lustre: server umount lustre-MDT0000 complete [ 4823.987337] LustreError: 75843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757396 with bad export cookie 5880244527069546851 [ 4823.992585] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4824.004026] LustreError: 75843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 11 previous similar messages [ 4824.022305] LustreError: Skipped 2 previous similar messages [ 4824.280480] Lustre: server umount lustre-MDT0001 complete [ 4840.042960] Lustre: server umount lustre-OST0000 complete [ 4844.514702] Lustre: 16402:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788757400/real 1788757400] req@ffffa0510a075880 x1875643515152384/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788757416 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4844.554456] Lustre: 16402:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 21 previous similar messages [ 4845.196425] Lustre: server umount lustre-OST0001 complete [ 4858.044234] Lustre: DEBUG MARKER: == sanity-scrub test 15: Dryrun mode OI scrub ============ 01:03:49 (1788757429) [ 4877.540902] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 4885.649536] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4932.527472] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 4944.326449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4944.719326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4944.783319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4944.901268] Lustre: lustre-MDT0000: new disk, initializing [ 4944.994240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4944.999214] Lustre: Skipped 7 previous similar messages [ 4945.024839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4951.222758] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4962.300656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4962.459149] Lustre: 139529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4962.473128] Lustre: 139529:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 4962.500928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4962.505644] Lustre: Skipped 1 previous similar message [ 4962.611281] Lustre: lustre-MDT0001: new disk, initializing [ 4962.732682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4962.745935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4968.940799] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4975.713407] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4984.877726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4985.050813] Lustre: lustre-OST0000: new disk, initializing [ 4985.053874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4985.058694] Lustre: 141162:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4987.098553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4987.109274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4987.256294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4993.549348] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5004.835382] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5005.008757] Lustre: lustre-OST0001: new disk, initializing [ 5005.016450] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5005.031366] Lustre: 142035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5006.321079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5006.329781] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5006.431337] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5013.402414] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5023.685809] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5028.167760] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5051.305896] Lustre: Failing over lustre-MDT0000 [ 5051.593419] Lustre: server umount lustre-MDT0000 complete [ 5052.389221] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.416637] Lustre: Skipped 11 previous similar messages [ 5055.782957] LustreError: 139522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757628 with bad export cookie 5880244527069666768 [ 5055.784237] Lustre: Failing over lustre-MDT0001 [ 5055.802121] LustreError: 139522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5056.413601] Lustre: server umount lustre-MDT0001 complete [ 5062.545654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5072.744401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5081.466816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5092.103557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5105.628353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5105.679879] Lustre: lustre-MDT0000: reset Object Index mappings [ 5105.691607] Lustre: Skipped 3 previous similar messages [ 5126.050534] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x519ad33e146c5f23 [ 5130.627783] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5141.230606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5141.543687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 5141.554341] LustreError: Skipped 8 previous similar messages [ 5141.708130] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:42 to 0x2c0000400:65) [ 5141.724211] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:65) [ 5142.811076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 5142.811179] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:65) [ 5146.853488] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5203.607320] Lustre: DEBUG MARKER: == sanity-scrub test 16: Initial OI scrub can rebuild crashed index objects ========================================================== 01:09:35 (1788757775) [ 5220.720325] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5269.934183] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5300.970360] Lustre: Failing over lustre-MDT0000 [ 5301.013827] Lustre: *** cfs_fail_loc=199, val=0*** [ 5301.022219] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000001:0x3:0x0]: rc = 0 [ 5301.040135] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2:0x0]: rc = 0 [ 5301.046776] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x9:0x0]: rc = 0 [ 5301.054587] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xa:0x0]: rc = 0 [ 5301.066443] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xb:0x0]: rc = 0 [ 5301.072492] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xc:0x0]: rc = 0 [ 5301.078469] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xd:0x0]: rc = 0 [ 5301.096047] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xe:0x0]: rc = 0 [ 5301.112400] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x13:0x0]: rc = 0 [ 5301.128996] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x14:0x0]: rc = 0 [ 5301.143722] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x15:0x0]: rc = 0 [ 5301.164296] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x16:0x0]: rc = 0 [ 5301.177546] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x17:0x0]: rc = 0 [ 5301.199139] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x18:0x0]: rc = 0 [ 5301.209756] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x19:0x0]: rc = 0 [ 5301.234588] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1a:0x0]: rc = 0 [ 5301.252419] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1b:0x0]: rc = 0 [ 5301.287663] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1c:0x0]: rc = 0 [ 5301.307531] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1d:0x0]: rc = 0 [ 5301.334982] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1e:0x0]: rc = 0 [ 5301.353868] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1f:0x0]: rc = 0 [ 5301.364087] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x20:0x0]: rc = 0 [ 5301.372031] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x21:0x0]: rc = 0 [ 5301.384576] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x22:0x0]: rc = 0 [ 5301.401617] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x23:0x0]: rc = 0 [ 5301.422127] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x25:0x0]: rc = 0 [ 5301.441919] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x26:0x0]: rc = 0 [ 5301.458762] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x27:0x0]: rc = 0 [ 5301.475168] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x28:0x0]: rc = 0 [ 5301.486745] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x29:0x0]: rc = 0 [ 5301.502170] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2a:0x0]: rc = 0 [ 5301.524602] Lustre: *** cfs_fail_loc=199, val=0*** [ 5301.530125] Lustre: Skipped 30 previous similar messages [ 5301.534416] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2b:0x0]: rc = 0 [ 5301.544649] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2c:0x0]: rc = 0 [ 5301.557334] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2d:0x0]: rc = 0 [ 5301.570316] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2e:0x0]: rc = 0 [ 5301.587078] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2f:0x0]: rc = 0 [ 5301.600431] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x30:0x0]: rc = 0 [ 5301.612420] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x31:0x0]: rc = 0 [ 5301.623121] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x32:0x0]: rc = 0 [ 5301.645617] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x33:0x0]: rc = 0 [ 5301.668635] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x34:0x0]: rc = 0 [ 5301.677328] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x35:0x0]: rc = 0 [ 5301.690290] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x1:0x0]: rc = 0 [ 5301.701555] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x2:0x0]: rc = 0 [ 5301.713361] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x3:0x0]: rc = 0 [ 5301.730403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5301.736668] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20001:0x0]: rc = 0 [ 5301.750040] Lustre: Skipped 4 previous similar messages [ 5301.772667] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20002:0x0]: rc = 0 [ 5301.788571] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20003:0x0]: rc = 0 [ 5301.808649] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x10000:0x0]: rc = 0 [ 5301.819673] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x20000:0x0]: rc = 0 [ 5301.829058] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x1010000:0x0]: rc = 0 [ 5301.843560] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x1020000:0x0]: rc = 0 [ 5301.865655] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x2010000:0x0]: rc = 0 [ 5301.884633] Lustre: 151784:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x2020000:0x0]: rc = 0 [ 5302.442467] Lustre: server umount lustre-MDT0000 complete [ 5306.896710] Lustre: Failing over lustre-MDT0001 [ 5306.898921] LustreError: 139523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757879 with bad export cookie 5880244527069683491 [ 5306.913981] Lustre: *** cfs_fail_loc=199, val=0*** [ 5306.923646] LustreError: 139523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5306.940660] Lustre: Skipped 22 previous similar messages [ 5306.943975] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000001:0x3:0x0]: rc = 0 [ 5306.953569] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x3:0x0]: rc = 0 [ 5306.961898] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x4:0x0]: rc = 0 [ 5306.970253] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x5:0x0]: rc = 0 [ 5306.978827] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x6:0x0]: rc = 0 [ 5306.984522] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x7:0x0]: rc = 0 [ 5306.990960] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x8:0x0]: rc = 0 [ 5306.998545] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xd:0x0]: rc = 0 [ 5307.004858] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xe:0x0]: rc = 0 [ 5307.016914] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xf:0x0]: rc = 0 [ 5307.023544] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x10:0x0]: rc = 0 [ 5307.038365] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x11:0x0]: rc = 0 [ 5307.045594] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x12:0x0]: rc = 0 [ 5307.055551] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x13:0x0]: rc = 0 [ 5307.067972] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x14:0x0]: rc = 0 [ 5307.075981] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x15:0x0]: rc = 0 [ 5307.084498] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x16:0x0]: rc = 0 [ 5307.093774] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x17:0x0]: rc = 0 [ 5307.100375] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x18:0x0]: rc = 0 [ 5307.113281] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x19:0x0]: rc = 0 [ 5307.121403] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1a:0x0]: rc = 0 [ 5307.134767] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1b:0x0]: rc = 0 [ 5307.151966] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1c:0x0]: rc = 0 [ 5307.168539] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1d:0x0]: rc = 0 [ 5307.175361] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1f:0x0]: rc = 0 [ 5307.184568] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x20:0x0]: rc = 0 [ 5307.194956] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x21:0x0]: rc = 0 [ 5307.205463] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x22:0x0]: rc = 0 [ 5307.213738] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x23:0x0]: rc = 0 [ 5307.218971] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x24:0x0]: rc = 0 [ 5307.226727] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x25:0x0]: rc = 0 [ 5307.235066] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x26:0x0]: rc = 0 [ 5307.251219] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x27:0x0]: rc = 0 [ 5307.258274] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x28:0x0]: rc = 0 [ 5307.266243] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x29:0x0]: rc = 0 [ 5307.280037] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2a:0x0]: rc = 0 [ 5307.293534] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2b:0x0]: rc = 0 [ 5307.308335] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2c:0x0]: rc = 0 [ 5307.323938] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2d:0x0]: rc = 0 [ 5307.333559] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2e:0x0]: rc = 0 [ 5307.345366] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2f:0x0]: rc = 0 [ 5307.363875] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x1:0x0]: rc = 0 [ 5307.374523] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x2:0x0]: rc = 0 [ 5307.381336] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x3:0x0]: rc = 0 [ 5307.391332] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20001:0x0]: rc = 0 [ 5307.406129] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20002:0x0]: rc = 0 [ 5307.413526] Lustre: 151984:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20003:0x0]: rc = 0 [ 5307.757536] Lustre: server umount lustre-MDT0001 complete [ 5318.769560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5318.916894] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5318.921994] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace' with [0x200000003:0x13:0x0]: rc = 0 [ 5318.932503] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000003:0xc:0x0]: rc = 0 [ 5318.941352] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5318.948123] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xa:0x0]: rc = 0 [ 5318.953824] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0x9:0x0]: rc = 0 [ 5318.962858] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000-MDT0000' with [0x200000005:0x20003:0x0]: rc = 0 [ 5318.969963] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000003:0xd:0x0]: rc = 0 [ 5318.974379] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000-MDT0000' with [0x200000005:0x20001:0x0]: rc = 0 [ 5318.981375] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000003:0xe:0x0]: rc = 0 [ 5318.987559] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5318.994410] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000-MDT0000' with [0x200000005:0x20002:0x0]: rc = 0 [ 5319.001624] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5319.009589] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xb:0x0]: rc = 0 [ 5319.019273] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_01' with [0x200000003:0x26:0x0]: rc = 0 [ 5319.027687] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_09' with [0x200000003:0x2e:0x0]: rc = 0 [ 5319.039882] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_00' with [0x200000003:0x25:0x0]: rc = 0 [ 5319.049319] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_02' with [0x200000003:0x27:0x0]: rc = 0 [ 5319.056274] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_09' with [0x200000003:0x1d:0x0]: rc = 0 [ 5319.066559] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_14' with [0x200000003:0x33:0x0]: rc = 0 [ 5319.075643] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_05' with [0x200000003:0x19:0x0]: rc = 0 [ 5319.084839] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_06' with [0x200000003:0x2b:0x0]: rc = 0 [ 5319.092309] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_08' with [0x200000003:0x1c:0x0]: rc = 0 [ 5319.099630] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_10' with [0x200000003:0x1e:0x0]: rc = 0 [ 5319.107842] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_13' with [0x200000003:0x32:0x0]: rc = 0 [ 5319.115422] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_02' with [0x200000003:0x16:0x0]: rc = 0 [ 5319.123222] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_00' with [0x200000003:0x14:0x0]: rc = 0 [ 5319.132659] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_10' with [0x200000003:0x2f:0x0]: rc = 0 [ 5319.144517] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_15' with [0x200000003:0x34:0x0]: rc = 0 [ 5319.153617] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_04' with [0x200000003:0x29:0x0]: rc = 0 [ 5319.164963] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_11' with [0x200000003:0x30:0x0]: rc = 0 [ 5319.178796] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_13' with [0x200000003:0x21:0x0]: rc = 0 [ 5319.196582] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_11' with [0x200000003:0x1f:0x0]: rc = 0 [ 5319.203890] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_07' with [0x200000003:0x1b:0x0]: rc = 0 [ 5319.208788] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_12' with [0x200000003:0x31:0x0]: rc = 0 [ 5319.214184] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_06' with [0x200000003:0x1a:0x0]: rc = 0 [ 5319.219283] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_08' with [0x200000003:0x2d:0x0]: rc = 0 [ 5319.224901] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_07' with [0x200000003:0x2c:0x0]: rc = 0 [ 5319.232898] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_15' with [0x200000003:0x23:0x0]: rc = 0 [ 5319.237970] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_04' with [0x200000003:0x18:0x0]: rc = 0 [ 5319.245445] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_01' with [0x200000003:0x15:0x0]: rc = 0 [ 5319.251946] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_03' with [0x200000003:0x28:0x0]: rc = 0 [ 5319.262614] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_05' with [0x200000003:0x2a:0x0]: rc = 0 [ 5319.270047] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_14' with [0x200000003:0x22:0x0]: rc = 0 [ 5319.278139] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_12' with [0x200000003:0x20:0x0]: rc = 0 [ 5319.286882] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_03' with [0x200000003:0x17:0x0]: rc = 0 [ 5319.295453] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x2:0x0]: rc = 0 [ 5319.308896] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 5319.341592] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 5319.358654] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 5319.374339] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 5319.386366] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 5319.395405] Lustre: 152482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 5338.104626] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5347.948887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5348.030260] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace' with [0x200000003:0xd:0x0]: rc = 0 [ 5348.043305] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5348.054682] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_10' with [0x200000003:0x18:0x0]: rc = 0 [ 5348.068364] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_10' with [0x200000003:0x29:0x0]: rc = 0 [ 5348.082321] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_14' with [0x200000003:0x2d:0x0]: rc = 0 [ 5348.094336] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_00' with [0x200000003:0xe:0x0]: rc = 0 [ 5348.113930] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_02' with [0x200000003:0x21:0x0]: rc = 0 [ 5348.123812] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_08' with [0x200000003:0x16:0x0]: rc = 0 [ 5348.145687] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_03' with [0x200000003:0x11:0x0]: rc = 0 [ 5348.172806] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_01' with [0x200000003:0xf:0x0]: rc = 0 [ 5348.183175] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_08' with [0x200000003:0x27:0x0]: rc = 0 [ 5348.196779] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_12' with [0x200000003:0x2b:0x0]: rc = 0 [ 5348.220702] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_15' with [0x200000003:0x2e:0x0]: rc = 0 [ 5348.240973] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_09' with [0x200000003:0x28:0x0]: rc = 0 [ 5348.255323] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_05' with [0x200000003:0x24:0x0]: rc = 0 [ 5348.264749] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_06' with [0x200000003:0x25:0x0]: rc = 0 [ 5348.273496] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_07' with [0x200000003:0x15:0x0]: rc = 0 [ 5348.282483] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_14' with [0x200000003:0x1c:0x0]: rc = 0 [ 5348.296133] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_13' with [0x200000003:0x2c:0x0]: rc = 0 [ 5348.310413] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_13' with [0x200000003:0x1b:0x0]: rc = 0 [ 5348.327239] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_11' with [0x200000003:0x19:0x0]: rc = 0 [ 5348.345330] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_04' with [0x200000003:0x12:0x0]: rc = 0 [ 5348.355244] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_02' with [0x200000003:0x10:0x0]: rc = 0 [ 5348.376945] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_09' with [0x200000003:0x17:0x0]: rc = 0 [ 5348.387883] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_12' with [0x200000003:0x1a:0x0]: rc = 0 [ 5348.404089] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_00' with [0x200000003:0x1f:0x0]: rc = 0 [ 5348.424253] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_06' with [0x200000003:0x14:0x0]: rc = 0 [ 5348.437219] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_15' with [0x200000003:0x1d:0x0]: rc = 0 [ 5348.448956] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_07' with [0x200000003:0x26:0x0]: rc = 0 [ 5348.464112] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_05' with [0x200000003:0x13:0x0]: rc = 0 [ 5348.474119] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_03' with [0x200000003:0x22:0x0]: rc = 0 [ 5348.486069] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_04' with [0x200000003:0x23:0x0]: rc = 0 [ 5348.500551] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_11' with [0x200000003:0x2a:0x0]: rc = 0 [ 5348.517803] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_01' with [0x200000003:0x20:0x0]: rc = 0 [ 5348.531920] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x4:0x0]: rc = 0 [ 5348.541652] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1020000-MDT0001' with [0x200000005:0x20002:0x0]: rc = 0 [ 5348.553098] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2020000' with [0x200000003:0x8:0x0]: rc = 0 [ 5348.565279] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5348.588496] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5348.600492] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x5:0x0]: rc = 0 [ 5348.614656] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5348.625945] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2020000-MDT0001' with [0x200000005:0x20003:0x0]: rc = 0 [ 5348.650103] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x3:0x0]: rc = 0 [ 5348.673765] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x20000' with [0x200000003:0x6:0x0]: rc = 0 [ 5348.687413] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5348.704200] Lustre: 153228:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x20000-MDT0001' with [0x200000005:0x20001:0x0]: rc = 0 [ 5349.121795] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:106 to 0x2c0000400:129) [ 5349.140638] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:105 to 0x280000400:129) [ 5354.200218] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5354.576638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:129) [ 5354.580550] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 5367.570391] Lustre: DEBUG MARKER: == sanity-scrub test 17a: ENOSPC on OI insert shouldn't leak inodes ========================================================== 01:12:19 (1788757939) [ 5368.630789] Lustre: *** cfs_fail_loc=19d, val=0*** [ 5368.635754] Lustre: Skipped 123 previous similar messages [ 5370.675199] Lustre: Failing over lustre-MDT0000 [ 5370.920404] Lustre: server umount lustre-MDT0000 complete [ 5387.968794] LustreError: 152496:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5388.002704] LustreError: 152496:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 43 previous similar messages [ 5390.267294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5399.016021] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x519ad33e146cda9f [ 5399.340276] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5399.345896] Lustre: Skipped 2 previous similar messages [ 5399.427100] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5399.432645] Lustre: Skipped 2 previous similar messages [ 5404.336431] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5404.650240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5404.665251] Lustre: Skipped 13 previous similar messages [ 5404.682464] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5404.693381] Lustre: Skipped 2 previous similar messages [ 5404.749844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:161) [ 5404.751069] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:161) [ 5408.501600] Lustre: DEBUG MARKER: == sanity-scrub test 17b: ENOSPC on .. insertion shouldn't leak inodes ========================================================== 01:12:59 (1788757979) [ 5410.230156] Lustre: *** cfs_fail_loc=19e, val=0*** [ 5412.473466] Lustre: Failing over lustre-MDT0000 [ 5412.953015] Lustre: server umount lustre-MDT0000 complete [ 5430.678659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5430.862454] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5430.879362] LustreError: Skipped 3 previous similar messages [ 5436.446311] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:193) [ 5436.447011] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:193) [ 5436.880318] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5441.592365] Lustre: DEBUG MARKER: == sanity-scrub test 18: test mount -o resetoi to recreate OI files ========================================================== 01:13:32 (1788758012) [ 5465.912690] Lustre: Failing over lustre-MDT0000 [ 5466.371989] Lustre: server umount lustre-MDT0000 complete [ 5471.326194] Lustre: Failing over lustre-MDT0001 [ 5471.818525] Lustre: server umount lustre-MDT0001 complete [ 5488.133846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5488.672625] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788758044/real 1788758044] req@ffffa0522d928380 x1875643515620736/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788758060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5488.707434] Lustre: 16401:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [ 5501.595907] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5511.267755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5511.643200] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 5511.681964] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:193) [ 5516.916794] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:234 to 0x2c0000401:257) [ 5516.927073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:233 to 0x280000401:257) [ 5517.892953] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5526.709555] Lustre: Failing over lustre-MDT0000 [ 5527.209349] Lustre: server umount lustre-MDT0000 complete [ 5531.831738] Lustre: Failing over lustre-MDT0001 [ 5532.378974] Lustre: server umount lustre-MDT0001 complete [ 5542.289387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5542.311716] Lustre: lustre-MDT0000: reset Object Index mappings [ 5542.319183] Lustre: Skipped 1 previous similar message [ 5558.214333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5558.219646] Lustre: Skipped 11 previous similar messages [ 5563.531835] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5573.906955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5574.425183] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:225) [ 5574.449086] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:225) [ 5579.437662] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5579.853716] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:234 to 0x2c0000401:289) [ 5579.858770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:233 to 0x280000401:289) [ 5592.434834] Lustre: DEBUG MARKER: == sanity-scrub test 19: LFSCK can fix multiple linked files on OST ========================================================== 01:16:03 (1788758163) [ 5605.357522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5612.600173] Lustre: server umount lustre-MDT0000 complete [ 5617.291386] LustreError: 151528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758189 with bad export cookie 5880244527069745133 [ 5617.293898] LustreError: 159915:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x2000032e1:0x1:0x0]: rc = -5 [ 5617.305889] LustreError: 151528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5617.722601] Lustre: server umount lustre-MDT0001 complete [ 5632.407927] Lustre: server umount lustre-OST0000 complete [ 5647.082235] Lustre: server umount lustre-OST0001 complete [ 5656.275631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5666.796359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5682.400411] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5687.520476] LustreError: 162011:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.131@tcp: failed processing log, type 4: rc = -110 [ 5720.397766] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5726.798906] Lustre: Failing over lustre-OST0000 [ 5727.011750] Lustre: server umount lustre-OST0000 complete [ 5734.215478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5744.930222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5760.544734] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5765.664437] LustreError: 163538:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.131@tcp: failed processing log, type 4: rc = -110 [ 5799.708351] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5810.753703] Lustre: DEBUG MARKER: == sanity-scrub test 20: Don't trigger OI scrub for irreparable oi repeatedly ========================================================== 01:19:41 (1788758381) [ 5828.059779] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 5844.537749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.331611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:385) [ 5850.080258] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5860.457498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5861.042097] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:257) [ 5867.403175] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5870.965977] Lustre: 166451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5870.991969] Lustre: 166451:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 5890.038686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5895.672995] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:257) [ 5895.693711] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:234 to 0x2c0000401:321) [ 5898.353898] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5906.249965] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5912.617168] Lustre: *** cfs_fail_loc=193, val=0*** [ 5914.725593] Lustre: Failing over lustre-MDT0000 [ 5914.948588] Lustre: *** cfs_fail_loc=193, val=0*** [ 5915.182176] Lustre: server umount lustre-MDT0000 complete [ 5916.129655] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.138643] Lustre: Skipped 37 previous similar messages [ 5923.153339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5923.350042] Lustre: *** cfs_fail_loc=193, val=0*** [ 5927.775479] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5928.530795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:234 to 0x2c0000401:353) [ 5928.531796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:417) [ 5932.667700] Lustre: *** cfs_fail_loc=19f, val=0*** [ 5932.674415] Lustre: Skipped 3 previous similar messages [ 5932.690132] Lustre: *** cfs_fail_loc=19f, val=0*** [ 5932.695825] Lustre: lustre-MDT0000: trigger OI scrub by RPC for [0x200003ab1:0x2:0x0]/16843009 with flags 0x4a: rc = 0 [ 5932.696140] Lustre: Skipped 82 previous similar messages [ 5947.285565] Lustre: DEBUG MARKER: == sanity-scrub test 21: don't hang MDS recovery when failed to get update log ========================================================== 01:21:58 (1788758518) [ 5950.028959] Lustre: Failing over lustre-MDT0000 [ 5950.287770] Lustre: server umount lustre-MDT0000 complete [ 5954.050407] Lustre: Failing over lustre-MDT0001 [ 5954.458620] Lustre: server umount lustre-MDT0001 complete [ 5957.967376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5965.849746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5985.854086] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5989.609470] LustreError: 170191:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5989.619649] LustreError: 170191:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 113 previous similar messages [ 5995.780506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5995.989924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 5996.003786] LustreError: Skipped 8 previous similar messages [ 5997.662088] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:289) [ 5997.665190] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:289) [ 6000.914393] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6001.240595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:449) [ 6001.245700] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:234 to 0x2c0000401:385) [ 6007.780597] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 200 [ 6014.986238] Lustre: DEBUG MARKER: == sanity-scrub test 22: LFSCK can recreate or fix the LASTID on MDT/OST ========================================================== 01:23:06 (1788758586) [ 6021.603406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6021.609343] Lustre: Skipped 6 previous similar messages [ 6025.112606] Lustre: server umount lustre-MDT0000 complete [ 6029.730480] Lustre: server umount lustre-MDT0001 complete [ 6043.119474] Lustre: server umount lustre-OST0000 complete [ 6055.943289] Lustre: server umount lustre-OST0001 complete [ 6062.966674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6077.821244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.536894] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6093.194929] Lustre: Failing over lustre-MDT0000 [ 6093.206452] LustreError: 173342:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6093.212803] LustreError: 173342:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6093.218246] LustreError: 173342:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 6093.360777] Lustre: server umount lustre-MDT0000 complete [ 6099.944525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6116.129778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 6121.921367] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6132.255524] Lustre: DEBUG MARKER: === sanity-scrub: start setup 01:25:04 (1788758704) === [ 6134.780311] LustreError: 174981:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6134.792414] LustreError: 174981:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6134.807128] LustreError: 174981:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 6135.263586] Lustre: server umount lustre-MDT0000 complete [ 6170.379305] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_hostid [ 6179.405242] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6242.623231] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing load_modules_local [ 6257.814276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6258.202886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6258.244568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6258.342803] Lustre: lustre-MDT0000: new disk, initializing [ 6258.440950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6258.449585] Lustre: Skipped 11 previous similar messages [ 6258.472275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6264.412907] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6281.468142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.709154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6281.712940] Lustre: Skipped 1 previous similar message [ 6281.814753] Lustre: lustre-MDT0001: new disk, initializing [ 6281.935712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6281.943213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6287.226722] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6292.675913] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6303.623969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6304.000303] Lustre: lustre-OST0000: new disk, initializing [ 6304.005343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6304.036731] Lustre: 182285:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6305.759549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6305.774392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6305.940176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6312.585576] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6328.736455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6328.968050] Lustre: lustre-OST0001: new disk, initializing [ 6328.972137] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6328.980130] Lustre: 183308:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6331.012378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6331.024835] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6331.115676] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6336.512361] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6346.398718] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6350.038040] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6355.853131] Lustre: DEBUG MARKER: === sanity-scrub: finish setup 01:28:47 (1788758927) === [ 6357.698427] Lustre: DEBUG MARKER: == sanity-scrub test complete, duration 6023 sec ========= 01:28:49 (1788758929) [ 6359.726794] Lustre: DEBUG MARKER: === sanity-scrub: start cleanup 01:28:51 (1788758931) === [ 6363.906136] Lustre: DEBUG MARKER: === sanity-scrub: finish cleanup 01:28:55 (1788758935) === [ 6369.251511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6374.408484] Lustre: server umount lustre-MDT0000 complete [ 6383.936425] LustreError: 180337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758956 with bad export cookie 5880244527069771824 [ 6383.938413] LustreError: MGC192.168.201.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6383.948073] LustreError: 180337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 10 previous similar messages [ 6383.967553] LustreError: Skipped 6 previous similar messages [ 6384.242646] Lustre: server umount lustre-MDT0001 complete [ 6404.064561] Lustre: 16404:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788758960/real 1788758960] req@ffffa0521301f100 x1875643515947776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788758976 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6404.112652] Lustre: 16404:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 19 previous similar messages [ 6404.184704] Lustre: server umount lustre-OST0000 complete [ 6413.094817] Lustre: server umount lustre-OST0001 complete [ 6431.073534] Lustre: DEBUG MARKER: oleg131-server.virtnet: executing unload_modules_local [ 6433.907542] Key type lgssc unregistered [ 6434.183993] LNet: 186722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6434.190835] LNetError: 186722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6434.205601] LNet: Removed LNI 192.168.201.131@tcp [ 6435.183140] Key type .llcrypt unregistered [ 6435.185636] Key type ._llcrypt unregistered