[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 457997784 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002315] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.007348] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010117] LSM: Security Framework initializing [ 0.011032] Yama: becoming mindful. [ 0.011784] SELinux: Initializing. [ 0.012046] *** VALIDATE selinux *** [ 0.019807] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024558] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027116] *** VALIDATE tmpfs *** [ 0.028322] *** VALIDATE proc *** [ 0.029260] *** VALIDATE cgroup *** [ 0.030010] *** VALIDATE cgroup2 *** [ 0.032238] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035035] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.039535] debug: unmapping init [mem 0xffffffffb4c59000-0xffffffffb4c60fff] [ 0.041205] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042738] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043029] ... version: 2 [ 0.044017] ... bit width: 48 [ 0.045014] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047015] ... max period: 00007fffffffffff [ 0.048016] ... fixed-purpose events: 3 [ 0.049013] ... event mask: 000000070000000f [ 0.050320] rcu: Hierarchical SRCU implementation. [ 0.052477] smp: Bringing up secondary CPUs ... [ 0.053611] x86: Booting SMP configuration: [ 0.054030] .... node #0, CPUs: #1 #2 #3 [ 0.065111] smp: Brought up 1 node, 4 CPUs [ 0.067009] smpboot: Max logical packages: 1 [ 0.068008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.099449] node 0 deferred pages initialised in 29ms [ 0.103006] devtmpfs: initialized [ 0.104280] x86/mm: Memory block size: 128MB [ 0.106947] gcov: version magic: 0x41383552 [ 0.108570] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.109155] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.110314] pinctrl core: initialized pinctrl subsystem [ 0.111238] [ 0.111593] ************************************************************* [ 0.112028] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.113018] ** ** [ 0.114016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.115013] ** ** [ 0.116014] ** This means that this kernel is built to expose internal ** [ 0.117019] ** IOMMU data structures, which may compromise security on ** [ 0.118014] ** your system. ** [ 0.119010] ** ** [ 0.120013] ** If you see this message and you are not debugging the ** [ 0.121014] ** kernel, report this immediately to your vendor! ** [ 0.122008] ** ** [ 0.123010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.124008] ************************************************************* [ 0.125688] NET: Registered protocol family 16 [ 0.126440] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.127071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.128064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.129425] cpuidle: using governor menu [ 0.130000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.133525] PCI: Using configuration type 1 for base access [ 0.134093] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.140101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.141013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.142136] cryptd: max_cpu_qlen set to 1000 [ 0.144118] ACPI: Added _OSI(Module Device) [ 0.145015] ACPI: Added _OSI(Processor Device) [ 0.146012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.147012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.151140] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.155407] ACPI: Interpreter enabled [ 0.156050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.156824] ACPI: Using IOAPIC for interrupt routing [ 0.157104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.158425] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.168790] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.169051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.170036] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.171084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.173347] acpiphp: Slot [2] registered [ 0.174103] acpiphp: Slot [5] registered [ 0.175094] acpiphp: Slot [6] registered [ 0.176113] acpiphp: Slot [7] registered [ 0.177118] acpiphp: Slot [8] registered [ 0.178118] acpiphp: Slot [9] registered [ 0.179113] acpiphp: Slot [10] registered [ 0.180107] acpiphp: Slot [3] registered [ 0.181000] acpiphp: Slot [4] registered [ 0.181000] acpiphp: Slot [11] registered [ 0.182103] acpiphp: Slot [12] registered [ 0.183090] acpiphp: Slot [13] registered [ 0.184096] acpiphp: Slot [14] registered [ 0.186115] acpiphp: Slot [15] registered [ 0.187132] acpiphp: Slot [16] registered [ 0.189103] acpiphp: Slot [17] registered [ 0.190081] acpiphp: Slot [18] registered [ 0.191086] acpiphp: Slot [19] registered [ 0.193124] acpiphp: Slot [20] registered [ 0.194142] acpiphp: Slot [21] registered [ 0.196175] acpiphp: Slot [22] registered [ 0.198079] acpiphp: Slot [23] registered [ 0.199012] acpiphp: Slot [24] registered [ 0.200071] acpiphp: Slot [25] registered [ 0.201013] acpiphp: Slot [26] registered [ 0.202101] acpiphp: Slot [27] registered [ 0.203105] acpiphp: Slot [28] registered [ 0.204103] acpiphp: Slot [29] registered [ 0.205000] acpiphp: Slot [30] registered [ 0.205000] acpiphp: Slot [31] registered [ 0.207096] PCI host bridge to bus 0000:00 [ 0.208053] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.210015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.211000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.212025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.215031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.217040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.218197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.222099] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.225319] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.234755] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.240059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.242021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.244023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.246020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.250086] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.251513] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.253042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.256811] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.262014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.272756] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.276013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.280075] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.290015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.297017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.311017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.321263] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.334016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.342016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.364019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.374000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.382022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.388040] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.402016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.412286] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.419015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.426017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.453020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.466663] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.472019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.478020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.495028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.501094] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.506018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.519015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.531019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.540748] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.544499] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.547399] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.549364] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.551172] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.556057] iommu: Default domain type: Passthrough [ 0.557430] SCSI subsystem initialized [ 0.558093] ACPI: bus type USB registered [ 0.559065] usbcore: registered new interface driver usbfs [ 0.560090] usbcore: registered new interface driver hub [ 0.562059] usbcore: registered new device driver usb [ 0.563114] pps_core: LinuxPPS API ver. 1 registered [ 0.564009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.566046] PTP clock support registered [ 0.568131] EDAC MC: Ver: 3.0.0 [ 0.570156] PCI: Using ACPI for IRQ routing [ 0.571650] NetLabel: Initializing [ 0.573013] NetLabel: domain hash size = 128 [ 0.574011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.576081] NetLabel: unlabeled traffic allowed by default [ 0.579038] vgaarb: loaded [ 0.580216] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.581010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.589113] clocksource: Switched to clocksource kvm-clock [ 0.694915] VFS: Disk quotas dquot_6.6.0 [ 0.696462] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.698778] *** VALIDATE ramfs *** [ 0.699890] *** VALIDATE hugetlbfs *** [ 0.701364] pnp: PnP ACPI init [ 0.703716] pnp: PnP ACPI: found 6 devices [ 0.718798] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.721989] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.724241] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.726424] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.728827] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.731321] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.733871] NET: Registered protocol family 2 [ 0.736100] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.741262] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.744477] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.748922] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.751906] TCP: Hash tables configured (established 65536 bind 65536) [ 0.754226] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.757285] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.759884] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.762372] NET: Registered protocol family 1 [ 0.764379] RPC: Registered named UNIX socket transport module. [ 0.766411] RPC: Registered udp transport module. [ 0.768084] RPC: Registered tcp transport module. [ 0.769779] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.772111] NET: Registered protocol family 44 [ 0.773708] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.775462] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.777120] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.778946] PCI: CLS 0 bytes, default 64 [ 0.780764] Unpacking initramfs... [ 2.191322] debug: unmapping init [mem 0xffff946cbcc54000-0xffff946cbffbffff] [ 2.195416] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.197505] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.200255] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.714828] Initialise system trusted keyrings [ 2.716576] Key type blacklist registered [ 2.718615] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.728098] zbud: loaded [ 2.731304] *** VALIDATE nfs *** [ 2.732185] *** VALIDATE nfs4 *** [ 2.733731] pstore: using deflate compression [ 2.737616] Platform Keyring initialized [ 2.844758] NET: Registered protocol family 38 [ 2.846478] Key type asymmetric registered [ 2.847727] Asymmetric key parser 'x509' registered [ 2.848887] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.851149] io scheduler mq-deadline registered [ 2.852785] io scheduler kyber registered [ 2.854583] io scheduler bfq registered [ 2.856400] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.859449] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.862017] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.864525] ACPI: Power Button [PWRF] [ 2.869673] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.876187] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.889895] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.896600] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.912739] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.939821] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.967302] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.971565] Non-volatile memory driver v1.3 [ 2.973310] Linux agpgart interface v0.103 [ 3.004550] virtio_blk virtio1: [vda] 146856 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.007436] vda: detected capacity change from 0 to 75190272 [ 3.021314] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.024281] vdb: detected capacity change from 0 to 1073741824 [ 3.038290] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.040671] vdc: detected capacity change from 0 to 2621440000 [ 3.055753] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.058777] vdd: detected capacity change from 0 to 2621440000 [ 3.071234] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.073194] vde: detected capacity change from 0 to 4294967296 [ 3.083797] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.085679] vdf: detected capacity change from 0 to 4294967296 [ 3.090106] libphy: Fixed MDIO Bus: probed [ 3.098215] usbcore: registered new interface driver usbserial_generic [ 3.100776] usbserial: USB Serial support registered for generic [ 3.103138] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.107283] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.108896] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.111103] mousedev: PS/2 mouse device common for all mice [ 3.114377] rtc_cmos 00:05: RTC can wake from S4 [ 3.117300] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.117625] rtc_cmos 00:05: registered as rtc0 [ 3.121163] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.121752] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.126481] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.127562] intel_pstate: CPU model not supported [ 3.132961] hid: raw HID events driver (C) Jiri Kosina [ 3.135094] usbcore: registered new interface driver usbhid [ 3.137199] usbhid: USB HID core driver [ 3.138702] drop_monitor: Initializing network drop monitor service [ 3.140981] Initializing XFRM netlink socket [ 3.142551] NET: Registered protocol family 10 [ 3.144473] Segment Routing with IPv6 [ 3.145832] NET: Registered protocol family 17 [ 3.148225] mpls_gso: MPLS GSO support [ 3.153226] RAS: Correctable Errors collector initialized. [ 3.155062] AVX version of gcm_enc/dec engaged. [ 3.156703] AES CTR mode by8 optimization enabled [ 3.224742] sched_clock: Marking stable (3224717845, 0)->(4061432254, -836714409) [ 3.227700] registered taskstats version 1 [ 3.229194] Loading compiled-in X.509 certificates [ 3.230529] zswap: loaded using pool lzo/zbud [ 3.251142] Key type big_key registered [ 3.261109] Key type encrypted registered [ 3.261900] ima: No TPM chip found, activating TPM-bypass! [ 3.262964] ima: Allocated hash algorithm: sha1 [ 3.263855] ima: No architecture policies found [ 3.264719] evm: Initialising EVM extended attributes: [ 3.265606] evm: security.selinux [ 3.266287] evm: security.ima [ 3.266841] evm: security.capability [ 3.267571] evm: HMAC attrs: 0x1 [ 3.269100] rtc_cmos 00:05: setting system clock to 2026-08-27 18:57:01 UTC (1787857021) [ 3.275558] debug: unmapping init [mem 0xffffffffb5c03000-0xffffffffb5dfffff] [ 3.278344] debug: unmapping init [mem 0xffffffffb4982000-0xffffffffb4c58fff] [ 3.282256] Write protecting the kernel read-only data: 28672k [ 3.285354] debug: unmapping init [mem 0xffffffffb3003000-0xffffffffb31fffff] [ 3.288652] debug: unmapping init [mem 0xffffffffb3914000-0xffffffffb39fffff] [ 3.318152] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.324439] systemd[1]: Detected virtualization kvm. [ 3.326237] systemd[1]: Detected architecture x86-64. [ 3.327880] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.353346] systemd[1]: No hostname configured. [ 3.354366] systemd[1]: Set hostname to . [ 3.355744] random: systemd: uninitialized urandom read (16 bytes read) [ 3.357113] systemd[1]: Initializing machine ID from random generator. [ 3.398198] random: ln: uninitialized urandom read (6 bytes read) [ 3.475326] random: systemd: uninitialized urandom read (16 bytes read) [ 3.477795] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.481756] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.489489] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.071083] device-mapper: uevent: version 1.0.3 [ 4.073588] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.718407] virtio_net virtio0 ens2: renamed from eth0 [ 4.733885] scsi host0: ata_piix [ 4.736345] random: fast init done [ 4.769656] scsi host1: ata_piix [ 4.771247] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.773774] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.200176] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.776135] random: crng init done [ 9.777621] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.125804] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.272382] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.499959] SELinux: Disabled at runtime. [ 11.557871] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.566375] systemd[1]: Detected virtualization kvm. [ 11.568452] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.109111] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.111306] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.115483] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.119215] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.122238] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.129282] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.133103] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ 12.200177] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.584697] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.002941] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.012893] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.164487] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.176253] EDAC sbridge: Ver: 1.1.2 [ 14.679916] Key type dns_resolver registered [ 14.990806] NFS: Registering the id_resolver key type [ 14.993085] Key type id_resolver registered [ 14.994838] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg238-server login: [ 41.666702] libcfs: loading out-of-tree module taints kernel. [ 41.687268] Key type ._llcrypt registered [ 41.688737] Key type .llcrypt registered [ 41.740678] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 54.946561] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 57.052301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.093741] alg: No test for adler32 (adler32-zlib) [ 58.904220] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 59.927855] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 61.695182] Key type lgssc registered [ 63.865669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.410084] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 95.875686] hrtimer: interrupt took 4073676 ns [ 121.855802] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 136.878555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 136.894120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.099703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 138.165145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 138.244614] Lustre: lustre-MDT0000: new disk, initializing [ 138.299500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.318961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 142.955620] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.549230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.664194] Lustre: 6479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 159.698829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 159.703036] Lustre: Skipped 1 previous similar message [ 159.788935] Lustre: lustre-MDT0001: new disk, initializing [ 159.825413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.837972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.844590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 163.474652] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.717593] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 179.756683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 180.148690] Lustre: lustre-OST0000: new disk, initializing [ 180.154359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 180.162393] Lustre: 8419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 180.302895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.365959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 185.377574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 185.429182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 187.402093] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.946936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.174267] Lustre: lustre-OST0001: new disk, initializing [ 203.183815] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 203.194979] Lustre: 9489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.317503] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.605662] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.006070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 211.013758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 211.084759] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 221.908875] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 229.307330] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 236.357864] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing check_logdir /tmp/testlogs/ [ 241.270386] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing yml_node [ 246.516291] Lustre: DEBUG MARKER: Client: 2.17.57.81 [ 249.576443] Lustre: DEBUG MARKER: MDS: 2.17.57.81 [ 252.141180] Lustre: DEBUG MARKER: OSS: 2.17.57.81 [ 254.010496] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-scrub ============----- Thu Aug 27 15:01:10 EDT 2026 [ 271.264572] Lustre: DEBUG MARKER: excepting tests: [ 281.876986] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 292.832060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.836799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 292.875459] Lustre: Skipped 1 previous similar message [ 297.628505] Lustre: server umount lustre-MDT0000 complete [ 297.951973] LustreError: 10316:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.970483] LustreError: 10316:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 303.085590] LustreError: 6492:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.108193] LustreError: 6492:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 307.433123] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857325 with bad export cookie 17101461717114274811 [ 307.442879] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 307.809590] Lustre: server umount lustre-MDT0001 complete [ 324.383188] Lustre: 3617:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857326/real 1787857326] req@ffff946d1612f0c0 x1874704023526912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857342 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 324.420738] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.440475] Lustre: Skipped 2 previous similar messages [ 327.747973] Lustre: server umount lustre-OST0000 complete [ 328.672926] Lustre: 3615:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857330/real 1787857330] req@ffff946d41e6de00 x1874704023527168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857346 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 329.759103] Lustre: 3618:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857331/real 1787857331] req@ffff946d1612d680 x1874704023527424/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 333.793054] Lustre: 3617:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857336/real 1787857336] req@ffff946d1612e1c0 x1874704023527808/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857352 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 337.809854] Lustre: server umount lustre-OST0001 complete [ 354.844956] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 358.416445] Key type lgssc unregistered [ 358.733927] LNet: 14782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.746021] LNetError: 14782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 359.782725] LNet: Removed LNI 192.168.202.138@tcp [ 360.962279] Key type .llcrypt unregistered [ 360.965659] Key type ._llcrypt unregistered [ 385.880935] Key type ._llcrypt registered [ 385.884689] Key type .llcrypt registered [ 386.062877] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 404.244280] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 405.898577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 406.022202] alg: No test for adler32 (adler32-zlib) [ 407.334921] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 407.700589] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 409.536537] Key type lgssc registered [ 411.553639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 465.353398] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 477.265403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 477.306415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.649894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 478.677438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 478.750535] Lustre: lustre-MDT0000: new disk, initializing [ 478.830351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.858598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 483.549730] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 496.774245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.882317] Lustre: 19237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 496.905488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 496.907866] Lustre: Skipped 1 previous similar message [ 496.955987] Lustre: lustre-MDT0001: new disk, initializing [ 497.008741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 497.029970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 497.050928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 500.974642] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 505.676842] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 514.691858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 514.918017] Lustre: lustre-OST0000: new disk, initializing [ 514.922944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 514.927871] Lustre: 21176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 514.996063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 518.422064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 518.431622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 518.506858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 520.521906] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 534.601334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 534.812203] Lustre: lustre-OST0001: new disk, initializing [ 534.820350] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 534.828675] Lustre: 22203:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 534.905646] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 541.461371] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 542.254967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 542.264143] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 542.313286] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 553.680828] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 561.538649] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 573.628744] Lustre: DEBUG MARKER: === sanity-scrub: finish setup 15:06:29 (1787857589) === [ 575.601323] Lustre: DEBUG MARKER: == sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 15:06:32 (1787857592) [ 596.526245] Lustre: Failing over lustre-MDT0000 [ 596.894934] Lustre: server umount lustre-MDT0000 complete [ 598.514192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.533949] Lustre: Skipped 3 previous similar messages [ 600.858567] LustreError: 21187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857619 with bad export cookie 16990437164794700304 [ 600.860757] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 600.866379] Lustre: Failing over lustre-MDT0001 [ 600.868159] LustreError: 21187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 601.189246] Lustre: server umount lustre-MDT0001 complete [ 611.986795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 618.975166] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857621/real 1787857621] req@ffff946c02d470c0 x1874704387410560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 619.018336] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.959937] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857623/real 1787857623] req@ffff946d084e7480 x1874704387410816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 620.986709] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 624.095688] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857626/real 1787857626] req@ffff946d084e52c0 x1874704387410944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 624.134756] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 626.207947] LustreError: 16398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff946d084e70c0 x1874704387412608/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 626.593574] LustreError: 21170:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.608433] LustreError: 21170:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 626.641533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.664207] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 627.615193] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857629/real 1787857629] req@ffff946c029dde00 x1874704387411456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857645 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 627.658220] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 629.737884] LustreError: 24220:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.754885] LustreError: 24220:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 632.151873] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 632.804554] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857635/real 1787857635] req@ffff946d084e6580 x1874704387411584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 632.841706] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 634.850771] LustreError: 24218:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.884854] LustreError: 24218:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 637.923980] LustreError: 24219:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.936390] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 637.942521] LustreError: 24219:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 641.492800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 641.757421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 641.873357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 641.916400] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:42 to 0x2c0000400:65) [ 641.923935] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:65) [ 647.149552] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 647.150168] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 647.170529] Lustre: Skipped 1 previous similar message [ 647.205242] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 647.223441] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 647.261673] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:65) [ 647.262309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 662.146404] Lustre: DEBUG MARKER: == sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 15:07:59 (1787857679) [ 681.635153] Lustre: Failing over lustre-MDT0000 [ 681.916828] Lustre: server umount lustre-MDT0000 complete [ 682.987481] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 682.991746] LustreError: 24220:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 683.002739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 683.005120] Lustre: Skipped 4 previous similar messages [ 683.076745] LustreError: 24220:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 687.181092] LustreError: 21187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857705 with bad export cookie 16990437164794716628 [ 687.208518] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 687.209917] Lustre: Failing over lustre-MDT0001 [ 687.719248] Lustre: server umount lustre-MDT0001 complete [ 699.612494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 704.287499] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857706/real 1787857706] req@ffff946d3f6d61c0 x1874704387532288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 704.316515] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 704.324091] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.111153] LustreError: 21171:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.259107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 713.325698] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 718.213283] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 720.294552] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857722/real 1787857722] req@ffff946d41006d00 x1874704387534464/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 720.324981] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 723.429901] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 723.443231] Lustre: Skipped 2 previous similar messages [ 727.275718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 727.560835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 727.713664] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:106 to 0x280000400:129) [ 727.714364] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:105 to 0x2c0000400:129) [ 732.476453] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 733.157848] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 733.163298] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 733.180606] Lustre: Skipped 1 previous similar message [ 733.226845] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 733.298546] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:106 to 0x2c0000401:129) [ 733.298702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 737.400359] Lustre: *** cfs_fail_loc=193, val=0*** [ 742.800975] Lustre: Failing over lustre-MDT0000 [ 743.045584] Lustre: server umount lustre-MDT0000 complete [ 743.404734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 743.410406] LustreError: 25728:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 743.436094] Lustre: Skipped 3 previous similar messages [ 743.462902] LustreError: 25728:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 10 previous similar messages [ 743.483372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 753.235314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 753.366259] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 753.627051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 753.638047] Lustre: Skipped 1 previous similar message [ 753.679046] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 758.782443] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 758.783825] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 758.790749] Lustre: Skipped 2 previous similar messages [ 758.840987] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 758.877303] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:106 to 0x2c0000401:161) [ 758.880766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:161) [ 759.021576] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 770.035487] Lustre: DEBUG MARKER: == sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 15:09:46 (1787857786) [ 786.075899] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 805.277476] Lustre: 30495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 830.590575] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 834.844648] Lustre: 31631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 849.698520] Lustre: *** cfs_fail_loc=198, val=0*** [ 862.175311] Lustre: Failing over lustre-MDT0000 [ 862.526208] Lustre: server umount lustre-MDT0000 complete [ 866.277749] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.281180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 866.284946] LustreError: 26613:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 866.284957] LustreError: 26613:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 866.289416] Lustre: Skipped 3 previous similar messages [ 866.581364] LustreError: 21187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857884 with bad export cookie 16990437164794745867 [ 866.584142] Lustre: Failing over lustre-MDT0001 [ 866.586947] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 866.600150] LustreError: 21187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 867.019644] Lustre: server umount lustre-MDT0001 complete [ 872.343825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 876.705717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 886.317797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 887.775178] Lustre: 16402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857889/real 1787857889] req@ffff946d374fed00 x1874704387708288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 887.802769] Lustre: 16402:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 887.814196] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.827928] Lustre: Skipped 1 previous similar message [ 890.912410] LustreError: 16398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff946d374fe580 x1874704387710080/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 891.667148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 891.760403] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 897.310365] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 906.216581] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 906.231080] Lustre: Skipped 3 previous similar messages [ 906.711664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 906.976765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 907.013926] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 907.147477] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:193) [ 907.180923] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:193) [ 912.125540] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 912.429641] Lustre: lustre-MDT0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 912.474367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:202 to 0x280000401:225) [ 912.478606] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:202 to 0x2c0000401:225) [ 926.227186] Lustre: DEBUG MARKER: == sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 15:12:22 (1787857942) [ 961.276398] Lustre: Failing over lustre-MDT0000 [ 961.570939] Lustre: server umount lustre-MDT0000 complete [ 963.552344] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.571339] LustreError: 33107:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 963.571488] Lustre: Skipped 3 previous similar messages [ 963.594159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 963.606088] LustreError: 33107:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 17 previous similar messages [ 966.259692] LustreError: 21187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857984 with bad export cookie 16990437164794773258 [ 966.261464] Lustre: Failing over lustre-MDT0001 [ 966.263797] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 966.275851] LustreError: 21187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 966.782287] Lustre: server umount lustre-MDT0001 complete [ 972.060620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 977.311813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 984.032347] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857986/real 1787857986] req@ffff946d247d3480 x1874704387842432/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787858002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 984.078735] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 990.707353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 990.925836] Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64 [ 1010.788042] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xebca28950a306c47 [ 1010.801455] Lustre: MGC192.168.202.138@tcp: Connection restored to 0@lo (at 0@lo) [ 1010.809226] Lustre: Skipped 4 previous similar messages [ 1011.080454] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1016.704957] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1027.111077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1027.176634] Lustre: lustre-MDT0001: invalid oi count 63, remove them, then set it to 64 [ 1027.450934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1027.608851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1027.618840] Lustre: Skipped 2 previous similar messages [ 1027.702042] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:233 to 0x280000400:257) [ 1027.706401] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:234 to 0x2c0000400:257) [ 1028.657056] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1028.716270] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1028.838938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:266 to 0x280000401:289) [ 1028.844939] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:265 to 0x2c0000401:289) [ 1033.278977] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1061.642198] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1082.598768] Lustre: 38936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1109.132362] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1113.786257] Lustre: 40072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1138.769044] Lustre: Failing over lustre-MDT0000 [ 1139.157312] Lustre: server umount lustre-MDT0000 complete [ 1141.743767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1141.757077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.774454] Lustre: Skipped 3 previous similar messages [ 1141.779096] LustreError: 25728:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1141.791852] LustreError: 25728:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 1143.368806] LustreError: 31652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858161 with bad export cookie 16990437164794801223 [ 1143.375711] Lustre: Failing over lustre-MDT0001 [ 1143.398649] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1143.952224] Lustre: server umount lustre-MDT0001 complete [ 1149.204877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1157.997759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1164.256220] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858166/real 1787858166] req@ffff946d374fe1c0 x1874704388000640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787858182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1164.286822] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 1171.373931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1171.434366] Lustre: lustre-MDT0000: invalid oi count 58, remove them, then set it to 64 [ 1187.935690] LustreError: 16398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff946d374fd2c0 x1874704388003328/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1188.367935] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1193.298543] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1202.096767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1202.195740] Lustre: lustre-MDT0001: invalid oi count 58, remove them, then set it to 64 [ 1202.856358] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:298 to 0x2c0000400:321) [ 1202.866183] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:297 to 0x280000400:321) [ 1207.788332] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1207.801162] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1207.821334] Lustre: Skipped 5 previous similar messages [ 1207.860806] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1207.969551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:330 to 0x280000401:353) [ 1207.971112] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:329 to 0x2c0000401:353) [ 1208.518351] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1236.502056] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1256.507303] Lustre: 44784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1282.155502] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1287.316767] Lustre: 45931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1311.010358] Lustre: Failing over lustre-MDT0000 [ 1311.205172] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.209289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.232275] Lustre: Skipped 8 previous similar messages [ 1311.261415] Lustre: Skipped 3 previous similar messages [ 1311.795148] Lustre: server umount lustre-MDT0000 complete [ 1316.867431] Lustre: Failing over lustre-MDT0001 [ 1316.896349] LustreError: 19230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858335 with bad export cookie 16990437164794828579 [ 1316.898341] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1316.912170] LustreError: 19230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1317.431042] Lustre: server umount lustre-MDT0001 complete [ 1323.041040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1330.792597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1342.747994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1342.878631] Lustre: lustre-MDT0000: invalid oi count 56, remove them, then set it to 64 [ 1362.401116] LustreError: 16398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff946c040d9a40 x1874704388155392/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1363.101605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.108037] Lustre: Skipped 2 previous similar messages [ 1363.142805] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1366.185840] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1366.197207] Lustre: Skipped 4 previous similar messages [ 1369.166708] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1379.990369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1380.059718] Lustre: lustre-MDT0001: invalid oi count 56, remove them, then set it to 64 [ 1380.363486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1380.380499] LustreError: Skipped 1 previous similar message [ 1380.591954] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:361 to 0x280000400:385) [ 1380.599339] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:362 to 0x2c0000400:385) [ 1382.572447] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1382.616497] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1382.701986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:393 to 0x280000401:417) [ 1382.703857] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:394 to 0x2c0000401:417) [ 1386.312566] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1405.527273] Lustre: DEBUG MARKER: == sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 15:20:22 (1787858422) [ 1424.077495] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1445.464511] Lustre: 50646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1472.993533] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1503.251692] Lustre: Failing over lustre-MDT0000 [ 1505.627537] Lustre: server umount lustre-MDT0000 complete [ 1505.762347] LustreError: 31645:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.776792] LustreError: 31645:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 27 previous similar messages [ 1510.088753] LustreError: 22202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858528 with bad export cookie 16990437164794855935 [ 1510.090496] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1510.095245] Lustre: Failing over lustre-MDT0001 [ 1510.099793] LustreError: 22202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1510.522958] Lustre: server umount lustre-MDT0001 complete [ 1516.768470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1527.199281] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858529/real 1787858529] req@ffff946d07a4f840 x1874704388310784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787858545 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1527.238446] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 17 previous similar messages [ 1527.339524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1536.739991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1550.209718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1564.360365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1564.381838] Lustre: lustre-MDT0000: reset Object Index mappings [ 1581.106674] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1586.327690] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1596.367881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1596.392958] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1596.399357] Lustre: Skipped 4 previous similar messages [ 1596.418967] Lustre: lustre-MDT0001: reset Object Index mappings [ 1596.816795] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:426 to 0x280000400:449) [ 1596.819457] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:425 to 0x2c0000400:449) [ 1597.796864] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1597.842773] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1597.897471] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:458 to 0x2c0000401:481) [ 1597.903943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:457 to 0x280000401:481) [ 1602.371696] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1615.990603] Lustre: DEBUG MARKER: == sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 15:23:52 (1787858632) [ 1643.223663] Lustre: Failing over lustre-MDT0000 [ 1643.565298] Lustre: server umount lustre-MDT0000 complete [ 1644.002183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1644.003482] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.008220] LustreError: Skipped 1 previous similar message [ 1644.017563] Lustre: Skipped 8 previous similar messages [ 1647.887956] LustreError: 19231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858666 with bad export cookie 16990437164794883291 [ 1647.890110] Lustre: Failing over lustre-MDT0001 [ 1647.891219] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1647.910749] LustreError: 19231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1648.412879] Lustre: server umount lustre-MDT0001 complete [ 1655.324791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1667.516340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1677.854394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1688.971552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1701.049232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1701.068533] Lustre: lustre-MDT0000: reset Object Index mappings [ 1718.754587] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xebca28950a321835 [ 1719.312979] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1724.633291] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1736.132796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1736.147561] Lustre: lustre-MDT0001: reset Object Index mappings [ 1736.451504] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:490 to 0x280000400:513) [ 1736.452226] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:489 to 0x2c0000400:513) [ 1741.794523] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1741.872523] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1741.968314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:521 to 0x280000401:545) [ 1741.968773] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:522 to 0x2c0000401:545) [ 1742.638951] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1752.386299] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004282:0x1:0x0]/32008: rc = 0 [ 1753.570823] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240003ab1:0x1:0x0]/64001 with flags 0x52: rc = 0 [ 1778.837614] Lustre: DEBUG MARKER: == sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 15:26:35 (1787858795) [ 1805.272773] Lustre: Failing over lustre-MDT0000 [ 1805.819526] Lustre: server umount lustre-MDT0000 complete [ 1810.444940] Lustre: Failing over lustre-MDT0001 [ 1810.447398] LustreError: 31652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858828 with bad export cookie 16990437164794910773 [ 1810.447398] LustreError: 19230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858828 with bad export cookie 16990437164794910773 [ 1810.447838] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1810.517035] LustreError: 19230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1810.836485] Lustre: server umount lustre-MDT0001 complete [ 1817.486856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1829.539919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1839.486822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1852.482738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1866.544813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1866.587047] Lustre: lustre-MDT0000: reset Object Index mappings [ 1880.032387] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xebca28950a328468 [ 1880.046831] Lustre: MGC192.168.202.138@tcp: Connection restored to 0@lo (at 0@lo) [ 1880.051020] Lustre: Skipped 10 previous similar messages [ 1880.392893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.398376] Lustre: Skipped 5 previous similar messages [ 1880.445559] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1886.169163] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1897.263587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1897.294096] Lustre: lustre-MDT0001: reset Object Index mappings [ 1897.759744] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:554 to 0x280000400:577) [ 1897.779220] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:553 to 0x2c0000400:577) [ 1898.739155] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1898.791410] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1898.867112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:585 to 0x280000401:609) [ 1898.867638] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:586 to 0x2c0000401:609) [ 1904.115255] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1914.622030] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004a52:0x1:0x0]/32029: rc = 0 [ 1917.864026] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004281:0x1:0x0]/32007 with flags 0x52: rc = 0 [ 2045.938182] Lustre: DEBUG MARKER: == sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ========================================================== 15:31:02 (1787859062) [ 2089.468117] Lustre: Failing over lustre-MDT0000 [ 2090.423444] Lustre: server umount lustre-MDT0000 complete [ 2093.535979] LustreError: 25728:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.585875] LustreError: 25728:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 24 previous similar messages [ 2094.965000] Lustre: Failing over lustre-MDT0001 [ 2094.966467] LustreError: 19232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859113 with bad export cookie 16990437164794938472 [ 2094.969895] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2095.035039] LustreError: 19232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2096.095045] Lustre: server umount lustre-MDT0001 complete [ 2102.995163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2115.039979] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859117/real 1787859117] req@ffff946c0da5d680 x1874704388773760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787859133 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2115.076896] Lustre: 16401:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 35 previous similar messages [ 2115.129847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2126.704102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2138.535306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2153.025512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2153.039897] Lustre: lustre-MDT0000: reset Object Index mappings [ 2165.896449] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2170.979472] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2183.152849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2183.489983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2183.501121] LustreError: Skipped 4 previous similar messages [ 2183.628132] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:617 to 0x2c0000400:641) [ 2183.642340] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:618 to 0x280000400:641) [ 2184.620699] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2184.696818] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2184.782117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:649 to 0x280000401:673) [ 2184.784237] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:650 to 0x2c0000401:673) [ 2190.239433] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2202.745036] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200005222:0x1:0x0]/63: rc = 0 [ 2206.066433] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004a51:0x1:0x0]/32029 with flags 0x52: rc = 0 [ 2306.540286] Lustre: DEBUG MARKER: == sanity-scrub test 4d: FID in LMA mismatch with object FID won't block create ========================================================== 15:35:23 (1787859323) [ 2323.024871] Lustre: *** cfs_fail_loc=19b, val=0*** [ 2323.035786] Lustre: Skipped 1 previous similar message [ 2323.539760] Lustre: *** cfs_fail_loc=19b, val=0*** [ 2323.542356] Lustre: Skipped 35 previous similar messages [ 2324.542691] Lustre: *** cfs_fail_loc=19b, val=0*** [ 2324.544529] Lustre: Skipped 229 previous similar messages [ 2346.464136] Lustre: DEBUG MARKER: == sanity-scrub test 4e: FID reuse can be fixed ========== 15:36:02 (1787859362) [ 2352.148138] Lustre: *** cfs_fail_loc=1a0, val=0*** [ 2352.155575] Lustre: Skipped 189 previous similar messages [ 2369.339235] Lustre: DEBUG MARKER: == sanity-scrub test 5: OI scrub state machine =========== 15:36:25 (1787859385) [ 2385.376930] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.386347] Lustre: Skipped 18 previous similar messages [ 2385.387669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2385.403207] Lustre: Skipped 1 previous similar message [ 2390.511066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2390.522691] Lustre: Skipped 3 previous similar messages [ 2391.718675] Lustre: server umount lustre-MDT0000 complete [ 2400.737443] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2402.702938] Lustre: server umount lustre-MDT0001 complete [ 2413.642190] Lustre: server umount lustre-OST0000 complete [ 2424.248339] Lustre: server umount lustre-OST0001 complete [ 2432.714061] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 2441.170111] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2499.297091] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2514.303849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2514.670859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2514.721575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2514.861139] Lustre: lustre-MDT0000: new disk, initializing [ 2515.025794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.035371] Lustre: Skipped 3 previous similar messages [ 2515.076735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2521.646519] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2535.456688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2535.613906] Lustre: 75858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2535.621657] Lustre: 75858:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2535.641355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2535.645937] Lustre: Skipped 1 previous similar message [ 2535.737895] Lustre: lustre-MDT0001: new disk, initializing [ 2535.930991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2535.944074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2541.434908] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2546.276980] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2554.545656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2554.789580] Lustre: lustre-OST0000: new disk, initializing [ 2554.795059] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2554.805671] Lustre: 77488:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2556.504851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2556.520229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2556.631475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2561.874587] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2575.006082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.098683] Lustre: lustre-OST0001: new disk, initializing [ 2575.101844] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2575.105815] Lustre: 78360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2576.726276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2576.735178] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2576.814339] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2583.263922] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2595.099538] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2599.281619] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2625.210424] Lustre: Failing over lustre-MDT0000 [ 2625.707528] Lustre: server umount lustre-MDT0000 complete [ 2630.463502] Lustre: Failing over lustre-MDT0001 [ 2630.471936] LustreError: 75850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859648 with bad export cookie 16990437164795076897 [ 2630.472554] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2630.487978] LustreError: 75850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2630.523299] LustreError: Skipped 1 previous similar message [ 2631.149435] Lustre: server umount lustre-MDT0001 complete [ 2637.755882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2650.557445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2660.266411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2672.503790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2686.300665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2686.326503] Lustre: lustre-MDT0000: reset Object Index mappings [ 2686.333774] Lustre: Skipped 1 previous similar message [ 2700.769771] LustreError: 16398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff946d3feed680 x1874704389116032/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2701.247120] LustreError: 77482:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.272792] LustreError: 77482:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 20 previous similar messages [ 2701.361241] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2705.764322] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2715.910408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2716.545058] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:42 to 0x2c0000400:65) [ 2716.546333] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:65) [ 2718.518327] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2718.520965] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2718.548106] Lustre: Skipped 10 previous similar messages [ 2718.597244] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2718.663916] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:65) [ 2718.667293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 2722.115287] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2734.180081] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000402:0x1:0x0]/32040: rc = 0 [ 2734.183661] Lustre: *** cfs_fail_loc=190, val=3*** [ 2735.280207] Lustre: *** cfs_fail_loc=190, val=3*** [ 2735.283809] Lustre: Skipped 1 previous similar message [ 2736.352542] Lustre: *** cfs_fail_loc=190, val=3*** [ 2737.543392] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240000402:0x1:0x0]/32002 with flags 0x52: rc = 0 [ 2739.359384] Lustre: *** cfs_fail_loc=190, val=3*** [ 2739.362264] Lustre: Skipped 2 previous similar messages [ 2743.711147] Lustre: *** cfs_fail_loc=190, val=3*** [ 2743.717359] Lustre: Skipped 2 previous similar messages [ 2750.654966] Lustre: Failing over lustre-MDT0000 [ 2750.999693] Lustre: server umount lustre-MDT0000 complete [ 2756.525652] Lustre: Failing over lustre-MDT0001 [ 2757.209830] Lustre: server umount lustre-MDT0001 complete [ 2769.461671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2776.031641] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859778/real 1787859778] req@ffff946d3d2c70c0 x1874704389156352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787859794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2776.076586] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 21 previous similar messages [ 2787.866385] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2799.010622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2799.346824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2799.362394] LustreError: Skipped 2 previous similar messages [ 2799.566304] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:42 to 0x2c0000400:97) [ 2799.567112] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:97) [ 2804.820640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:97) [ 2804.832349] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:97) [ 2805.239392] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2810.913492] Lustre: Failing over lustre-MDT0000 [ 2813.198224] Lustre: server umount lustre-MDT0000 complete [ 2818.774274] Lustre: Failing over lustre-MDT0001 [ 2819.315333] Lustre: server umount lustre-MDT0001 complete [ 2832.158445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2832.333031] Lustre: *** cfs_fail_loc=190, val=3*** [ 2832.337946] Lustre: Skipped 2 previous similar messages [ 2850.400753] Lustre: *** cfs_fail_loc=190, val=3*** [ 2850.406762] Lustre: Skipped 5 previous similar messages [ 2852.458262] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2864.160813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2864.671967] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:129) [ 2864.675729] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:42 to 0x2c0000400:129) [ 2869.845745] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:129) [ 2869.847196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:129) [ 2870.300938] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2884.711233] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240000402:0x1:0x0]/32002 with flags 0x52: rc = 0 [ 2884.737338] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000402:0x45:0x0]/284: rc = 0 [ 2899.958364] Lustre: DEBUG MARKER: == sanity-scrub test 6: OI scrub resumes from last checkpoint ========================================================== 15:45:16 (1787859916) [ 2932.566435] Lustre: Failing over lustre-MDT0000 [ 2932.916269] Lustre: server umount lustre-MDT0000 complete [ 2937.842083] Lustre: Failing over lustre-MDT0001 [ 2938.170884] Lustre: server umount lustre-MDT0001 complete [ 2945.132795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2957.113467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2968.240677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2981.072118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2996.638574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2996.689340] Lustre: lustre-MDT0000: reset Object Index mappings [ 2996.693314] Lustre: Skipped 1 previous similar message [ 3008.032315] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xebca28950a359bbc [ 3014.792993] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3024.747672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3025.299669] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3025.315423] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:193) [ 3026.401044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 3026.401177] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:170 to 0x2c0000401:193) [ 3031.380370] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3043.152898] Lustre: *** cfs_fail_loc=190, val=2*** [ 3043.153214] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200001b72:0x1:0x0]/32002: rc = 0 [ 3043.154143] Lustre: Skipped 14 previous similar messages [ 3046.485159] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240001b71:0x1:0x0]/32036 with flags 0x52: rc = 0 [ 3060.035927] Lustre: lustre-MDT0001: trigger partial OI scrub for RPC inconsistency, checking FID [0x240001b71:0x44:0x0]/226: rc = 0 [ 3060.047992] Lustre: Skipped 1 previous similar message [ 3076.963762] Lustre: Failing over lustre-MDT0000 [ 3077.366369] Lustre: server umount lustre-MDT0000 complete [ 3077.603386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.620249] Lustre: Skipped 28 previous similar messages [ 3081.366148] Lustre: Failing over lustre-MDT0001 [ 3081.731261] Lustre: server umount lustre-MDT0001 complete [ 3092.236820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3107.423630] Lustre: *** cfs_fail_loc=190, val=3*** [ 3107.424976] Lustre: Skipped 31 previous similar messages [ 3111.277601] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3120.356656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3120.754224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3120.756744] Lustre: Skipped 12 previous similar messages [ 3120.796495] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:225) [ 3120.801340] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:225) [ 3125.637723] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3126.315233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:225) [ 3126.317379] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:170 to 0x2c0000401:225) [ 3146.014324] Lustre: DEBUG MARKER: == sanity-scrub test 7: System is available during OI scrub scanning ========================================================== 15:49:22 (1787860162) [ 3166.461310] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3191.137338] Lustre: 96472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3220.061374] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3224.718167] Lustre: 97609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3268.793704] Lustre: Failing over lustre-MDT0000 [ 3269.307802] Lustre: server umount lustre-MDT0000 complete [ 3274.042064] LustreError: 75850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860292 with bad export cookie 16990437164795142963 [ 3274.046989] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3274.048819] LustreError: 75850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 12 previous similar messages [ 3274.058018] LustreError: Skipped 4 previous similar messages [ 3274.060091] Lustre: Failing over lustre-MDT0001 [ 3274.415195] Lustre: server umount lustre-MDT0001 complete [ 3281.023404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3292.742761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3301.471253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3311.130737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3323.441592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3323.464954] Lustre: lustre-MDT0000: reset Object Index mappings [ 3323.467193] Lustre: Skipped 1 previous similar message [ 3345.912022] LustreError: 88058:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.920325] LustreError: 88058:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 62 previous similar messages [ 3345.990702] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3345.993614] Lustre: Skipped 4 previous similar messages [ 3351.736443] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3359.215716] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3359.227907] Lustre: Skipped 25 previous similar messages [ 3361.301811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3361.748461] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:265 to 0x2c0000400:289) [ 3361.764222] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:266 to 0x280000400:289) [ 3366.884568] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3366.909745] Lustre: Skipped 4 previous similar messages [ 3366.951076] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3366.956317] Lustre: Skipped 4 previous similar messages [ 3367.008250] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:266 to 0x2c0000401:289) [ 3367.009373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 3367.462848] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3376.896567] Lustre: *** cfs_fail_loc=190, val=3*** [ 3376.897802] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200002b12:0x1:0x0]/32048: rc = 0 [ 3376.910340] Lustre: Skipped 13 previous similar messages [ 3380.220850] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240002b11:0x1:0x0]/32004 with flags 0x52: rc = 0 [ 3404.328121] Lustre: DEBUG MARKER: == sanity-scrub test 8: Control OI scrub manually ======== 15:53:40 (1787860420) [ 3456.553087] Lustre: Failing over lustre-MDT0000 [ 3456.872735] Lustre: server umount lustre-MDT0000 complete [ 3459.058589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3459.064129] LustreError: Skipped 7 previous similar messages [ 3461.086707] Lustre: Failing over lustre-MDT0001 [ 3461.510454] Lustre: server umount lustre-MDT0001 complete [ 3467.776124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3478.459603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3480.480967] Lustre: 16402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787860482/real 1787860482] req@ffff946d3aad7480 x1874704389672320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787860498 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3480.542500] Lustre: 16402:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 54 previous similar messages [ 3489.616061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3503.047049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3517.134835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3531.753542] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xebca28950a37c5db [ 3536.466696] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3545.875509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3546.469412] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:329 to 0x280000400:353) [ 3546.479363] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:330 to 0x2c0000400:353) [ 3549.565071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:329 to 0x280000401:353) [ 3549.581058] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:330 to 0x2c0000401:353) [ 3552.109158] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3602.174057] Lustre: DEBUG MARKER: == sanity-scrub test 9: OI scrub speed control =========== 15:56:58 (1787860618) [ 3620.015408] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3641.407614] Lustre: 109154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3669.489566] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3674.093561] Lustre: 110291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3809.938241] Lustre: Failing over lustre-MDT0000 [ 3810.546990] Lustre: server umount lustre-MDT0000 complete [ 3810.787056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.799305] Lustre: Skipped 17 previous similar messages [ 3815.096702] Lustre: Failing over lustre-MDT0001 [ 3815.762323] Lustre: server umount lustre-MDT0001 complete [ 3822.371359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3836.725934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3848.690624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3862.730138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3880.099633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3880.128941] Lustre: lustre-MDT0000: reset Object Index mappings [ 3880.131169] Lustre: Skipped 3 previous similar messages [ 3886.117369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3886.121120] Lustre: Skipped 4 previous similar messages [ 3891.729325] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3904.340450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3905.190360] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:393 to 0x280000400:417) [ 3905.193398] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:394 to 0x2c0000400:417) [ 3910.213881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:393 to 0x280000401:417) [ 3910.226519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:394 to 0x2c0000401:417) [ 3911.641894] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3980.449016] Lustre: DEBUG MARKER: == sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount (1) ========================================================== 16:03:16 (1787860996) [ 3996.172559] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4020.505128] Lustre: 117144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4049.976917] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4054.111514] Lustre: 118280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4247.507569] Lustre: Failing over lustre-MDT0000 [ 4247.870576] Lustre: server umount lustre-MDT0000 complete [ 4248.033964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4248.040540] LustreError: 113246:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4248.041786] LustreError: Skipped 3 previous similar messages [ 4248.068496] LustreError: 113246:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 28 previous similar messages [ 4252.131599] Lustre: Failing over lustre-MDT0001 [ 4252.132498] LustreError: 76661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861270 with bad export cookie 16990437164795493264 [ 4252.133519] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4252.133528] LustreError: Skipped 2 previous similar messages [ 4252.193894] LustreError: 76661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4252.524967] Lustre: server umount lustre-MDT0001 complete [ 4259.291427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4269.472566] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861271/real 1787861271] req@ffff946d04724000 x1874704390162688/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787861287 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4269.510727] Lustre: 16399:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 24 previous similar messages [ 4271.479608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4280.984366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4292.130453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4305.939140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4322.783593] LustreError: 16398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff946d04724f00 x1874704390166400/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4323.535941] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4323.557297] Lustre: Skipped 2 previous similar messages [ 4328.999267] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4338.148544] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4338.152956] Lustre: Skipped 15 previous similar messages [ 4339.699747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4340.239901] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:458 to 0x280000400:481) [ 4340.248180] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:457 to 0x2c0000400:481) [ 4341.233861] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4341.253298] Lustre: Skipped 2 previous similar messages [ 4341.297110] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4341.315900] Lustre: Skipped 2 previous similar messages [ 4341.423796] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:458 to 0x2c0000401:481) [ 4341.428069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:457 to 0x280000401:481) [ 4346.016849] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4355.196681] Lustre: *** cfs_fail_loc=190, val=1*** [ 4355.197019] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004282:0x1:0x0]/32004: rc = 0 [ 4355.199572] Lustre: Skipped 61 previous similar messages [ 4358.471842] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004281:0x1:0x0]/32025 with flags 0x52: rc = 0 [ 4366.206532] Lustre: Failing over lustre-MDT0000 [ 4368.472892] Lustre: server umount lustre-MDT0000 complete [ 4372.627295] Lustre: Failing over lustre-MDT0001 [ 4373.125879] Lustre: server umount lustre-MDT0001 complete [ 4383.941695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4397.345819] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xebca28950a45afd1 [ 4403.526984] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4413.581746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4414.085106] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:458 to 0x280000400:513) [ 4414.112327] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:457 to 0x2c0000400:513) [ 4419.153825] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:458 to 0x2c0000401:513) [ 4419.157963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:457 to 0x280000401:513) [ 4419.407661] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4426.270444] Lustre: Failing over lustre-MDT0000 [ 4426.512837] Lustre: server umount lustre-MDT0000 complete [ 4427.240783] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4427.263440] Lustre: Skipped 18 previous similar messages [ 4431.274051] Lustre: Failing over lustre-MDT0001 [ 4431.803116] Lustre: server umount lustre-MDT0001 complete [ 4441.818935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4441.956821] Lustre: *** cfs_fail_loc=190, val=1*** [ 4441.963501] Lustre: Skipped 26 previous similar messages [ 4455.909205] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xebca28950a45b630 [ 4460.217458] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4469.068645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4469.440327] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:457 to 0x2c0000400:545) [ 4469.451172] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:458 to 0x280000400:545) [ 4474.652886] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4475.004791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:457 to 0x280000401:545) [ 4475.012243] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:458 to 0x2c0000401:545) [ 4497.646618] Lustre: DEBUG MARKER: == sanity-scrub test 11: OI scrub skips the new created objects only once ========================================================== 16:11:53 (1787861513) [ 4515.583352] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4536.369300] Lustre: 128173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4562.638101] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4599.308074] Lustre: DEBUG MARKER: == sanity-scrub test 12: OI scrub can rebuild invalid /O entries ========================================================== 16:13:36 (1787861616) [ 4605.646563] Lustre: *** cfs_fail_loc=195, val=0*** [ 4606.213509] Lustre: *** cfs_fail_loc=195, val=0*** [ 4606.216293] Lustre: Skipped 31 previous similar messages [ 4611.399905] Lustre: Failing over lustre-OST0000 [ 4611.573762] Lustre: server umount lustre-OST0000 complete [ 4623.067706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4623.322226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4623.325218] Lustre: Skipped 7 previous similar messages [ 4630.987063] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4827.184890] Lustre: DEBUG MARKER: == sanity-scrub test 13: OI scrub can rebuild missed /O entries ========================================================== 16:17:23 (1787861843) [ 4831.854411] Lustre: *** cfs_fail_loc=196, val=0*** [ 4831.858697] Lustre: Skipped 31 previous similar messages [ 4839.492633] Lustre: Failing over lustre-OST0000 [ 4839.560581] Lustre: server umount lustre-OST0000 complete [ 4848.610143] LustreError: 88081:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.633311] LustreError: 88081:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 45 previous similar messages [ 4850.000125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4850.183871] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4858.153951] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5048.528661] Lustre: DEBUG MARKER: == sanity-scrub test 14: OI scrub can repair OST objects under lost+found ========================================================== 16:21:05 (1787862065) [ 5056.522711] Lustre: *** cfs_fail_loc=196, val=0*** [ 5056.527279] Lustre: Skipped 63 previous similar messages [ 5060.633568] Lustre: *** cfs_fail_loc=196, val=0*** [ 5060.639471] Lustre: Skipped 287 previous similar messages [ 5069.124915] Lustre: *** cfs_fail_loc=196, val=0*** [ 5069.131029] Lustre: Skipped 511 previous similar messages [ 5083.014868] Lustre: Failing over lustre-OST0000 [ 5083.298732] Lustre: server umount lustre-OST0000 complete [ 5085.665768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.689831] Lustre: Skipped 6 previous similar messages [ 5098.440478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5099.180927] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5099.203057] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5099.219039] Lustre: Skipped 4 previous similar messages [ 5100.580736] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5100.596585] Lustre: Skipped 4 previous similar messages [ 5100.629845] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5100.633194] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5100.642828] Lustre: Skipped 4 previous similar messages [ 5100.676567] Lustre: Skipped 20 previous similar messages [ 5107.761404] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5119.973979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5119.982697] Lustre: Skipped 5 previous similar messages [ 5119.986580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5119.998748] LustreError: Skipped 5 previous similar messages [ 5122.421418] Lustre: server umount lustre-MDT0000 complete [ 5127.093985] LustreError: 75848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862145 with bad export cookie 16990437164796196400 [ 5127.103363] LustreError: 75848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5127.119914] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5127.137545] LustreError: Skipped 2 previous similar messages [ 5127.647807] Lustre: server umount lustre-MDT0001 complete [ 5142.766112] Lustre: server umount lustre-OST0000 complete [ 5146.594480] Lustre: 16402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862148/real 1787862148] req@ffff946d10190780 x1874704390667648/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787862164 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5146.622587] Lustre: 16402:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 32 previous similar messages [ 5147.434838] Lustre: server umount lustre-OST0001 complete [ 5161.557253] Lustre: DEBUG MARKER: == sanity-scrub test 15: Dryrun mode OI scrub ============ 16:22:57 (1787862177) [ 5181.457608] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5191.335678] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5251.352190] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5264.434753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5264.724357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5264.744878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5264.798231] Lustre: lustre-MDT0000: new disk, initializing [ 5264.848703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5264.865849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5270.815380] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5284.923408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5285.047486] Lustre: 139503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5285.066295] Lustre: 139503:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 5285.119914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5285.129258] Lustre: Skipped 1 previous similar message [ 5285.310334] Lustre: lustre-MDT0001: new disk, initializing [ 5285.467535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5285.512044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5290.127719] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5295.185818] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5302.822680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5303.155779] Lustre: lustre-OST0000: new disk, initializing [ 5303.161694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5303.173385] Lustre: 141134:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5305.079932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5305.114473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5305.173769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5310.685587] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5323.061483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5323.183215] Lustre: lustre-OST0001: new disk, initializing [ 5323.185403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5323.191373] Lustre: 142005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5324.604770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5324.613273] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5324.772229] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5332.092361] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5344.346799] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5349.206329] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5373.735341] Lustre: Failing over lustre-MDT0000 [ 5374.243633] Lustre: server umount lustre-MDT0000 complete [ 5378.717281] Lustre: Failing over lustre-MDT0001 [ 5379.106676] Lustre: server umount lustre-MDT0001 complete [ 5385.903515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5397.247646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5407.403975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5418.225216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5431.906745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5431.928798] Lustre: lustre-MDT0000: reset Object Index mappings [ 5431.932931] Lustre: Skipped 3 previous similar messages [ 5448.671669] LustreError: 16398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff946c057bc780 x1874704390792704/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5448.962821] LustreError: 141128:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5448.992770] LustreError: 141128:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 14 previous similar messages [ 5453.692565] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5467.917059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5468.450639] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:42 to 0x2c0000400:65) [ 5468.450838] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:65) [ 5473.904469] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:65) [ 5473.905359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 5474.304320] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5537.697651] Lustre: DEBUG MARKER: == sanity-scrub test 16: Initial OI scrub can rebuild crashed index objects ========================================================== 16:29:13 (1787862553) [ 5556.305620] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5578.140906] Lustre: 150302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5605.849060] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5610.661472] Lustre: 151437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5643.145295] Lustre: Failing over lustre-MDT0000 [ 5643.165822] Lustre: *** cfs_fail_loc=199, val=0*** [ 5643.178218] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000001:0x3:0x0]: rc = 0 [ 5643.201780] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2:0x0]: rc = 0 [ 5643.216704] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x9:0x0]: rc = 0 [ 5643.224987] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xa:0x0]: rc = 0 [ 5643.237800] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xb:0x0]: rc = 0 [ 5643.252491] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xc:0x0]: rc = 0 [ 5643.261712] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xd:0x0]: rc = 0 [ 5643.277695] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xe:0x0]: rc = 0 [ 5643.292588] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x13:0x0]: rc = 0 [ 5643.316766] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x14:0x0]: rc = 0 [ 5643.327725] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x15:0x0]: rc = 0 [ 5643.337939] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x16:0x0]: rc = 0 [ 5643.356817] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x17:0x0]: rc = 0 [ 5643.368149] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x18:0x0]: rc = 0 [ 5643.382238] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x19:0x0]: rc = 0 [ 5643.409816] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1a:0x0]: rc = 0 [ 5643.455984] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1b:0x0]: rc = 0 [ 5643.473722] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1c:0x0]: rc = 0 [ 5643.483736] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1d:0x0]: rc = 0 [ 5643.494977] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1e:0x0]: rc = 0 [ 5643.504743] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1f:0x0]: rc = 0 [ 5643.513867] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x20:0x0]: rc = 0 [ 5643.521752] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x21:0x0]: rc = 0 [ 5643.534705] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x22:0x0]: rc = 0 [ 5643.545041] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x23:0x0]: rc = 0 [ 5643.556878] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x25:0x0]: rc = 0 [ 5643.591254] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x26:0x0]: rc = 0 [ 5643.601835] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x27:0x0]: rc = 0 [ 5643.626710] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x28:0x0]: rc = 0 [ 5643.651545] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x29:0x0]: rc = 0 [ 5643.663996] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2a:0x0]: rc = 0 [ 5643.685327] Lustre: *** cfs_fail_loc=199, val=0*** [ 5643.690297] Lustre: Skipped 30 previous similar messages [ 5643.696303] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2b:0x0]: rc = 0 [ 5643.710819] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2c:0x0]: rc = 0 [ 5643.721494] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2d:0x0]: rc = 0 [ 5643.739468] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2e:0x0]: rc = 0 [ 5643.757854] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2f:0x0]: rc = 0 [ 5643.768800] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x30:0x0]: rc = 0 [ 5643.781661] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x31:0x0]: rc = 0 [ 5643.794152] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x32:0x0]: rc = 0 [ 5643.809785] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x33:0x0]: rc = 0 [ 5643.838757] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x34:0x0]: rc = 0 [ 5643.861797] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x35:0x0]: rc = 0 [ 5643.877398] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x1:0x0]: rc = 0 [ 5643.891973] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x2:0x0]: rc = 0 [ 5643.908961] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x3:0x0]: rc = 0 [ 5643.921281] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20001:0x0]: rc = 0 [ 5643.938426] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20002:0x0]: rc = 0 [ 5643.958170] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20003:0x0]: rc = 0 [ 5643.976549] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x10000:0x0]: rc = 0 [ 5643.989077] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x20000:0x0]: rc = 0 [ 5644.000465] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x1010000:0x0]: rc = 0 [ 5644.017356] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x1020000:0x0]: rc = 0 [ 5644.034621] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x2010000:0x0]: rc = 0 [ 5644.062171] Lustre: 151768:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x2020000:0x0]: rc = 0 [ 5644.450227] Lustre: server umount lustre-MDT0000 complete [ 5648.511414] Lustre: Failing over lustre-MDT0001 [ 5648.525210] Lustre: *** cfs_fail_loc=199, val=0*** [ 5648.531255] Lustre: Skipped 22 previous similar messages [ 5648.537624] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000001:0x3:0x0]: rc = 0 [ 5648.548240] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x3:0x0]: rc = 0 [ 5648.555245] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x4:0x0]: rc = 0 [ 5648.563565] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x5:0x0]: rc = 0 [ 5648.571443] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x6:0x0]: rc = 0 [ 5648.579995] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x7:0x0]: rc = 0 [ 5648.585887] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x8:0x0]: rc = 0 [ 5648.593903] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xd:0x0]: rc = 0 [ 5648.605109] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xe:0x0]: rc = 0 [ 5648.613708] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xf:0x0]: rc = 0 [ 5648.619111] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x10:0x0]: rc = 0 [ 5648.628192] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x11:0x0]: rc = 0 [ 5648.647979] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x12:0x0]: rc = 0 [ 5648.656810] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x13:0x0]: rc = 0 [ 5648.663497] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x14:0x0]: rc = 0 [ 5648.669766] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x15:0x0]: rc = 0 [ 5648.677990] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x16:0x0]: rc = 0 [ 5648.692668] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x17:0x0]: rc = 0 [ 5648.708499] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x18:0x0]: rc = 0 [ 5648.720600] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x19:0x0]: rc = 0 [ 5648.731525] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1a:0x0]: rc = 0 [ 5648.739971] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1b:0x0]: rc = 0 [ 5648.748833] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1c:0x0]: rc = 0 [ 5648.762980] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1d:0x0]: rc = 0 [ 5648.772825] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1f:0x0]: rc = 0 [ 5648.780611] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x20:0x0]: rc = 0 [ 5648.791863] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x21:0x0]: rc = 0 [ 5648.803660] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x22:0x0]: rc = 0 [ 5648.811759] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x23:0x0]: rc = 0 [ 5648.820371] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x24:0x0]: rc = 0 [ 5648.830690] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x25:0x0]: rc = 0 [ 5648.844170] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x26:0x0]: rc = 0 [ 5648.853093] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x27:0x0]: rc = 0 [ 5648.861116] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x28:0x0]: rc = 0 [ 5648.867341] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x29:0x0]: rc = 0 [ 5648.874703] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2a:0x0]: rc = 0 [ 5648.881561] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2b:0x0]: rc = 0 [ 5648.906162] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2c:0x0]: rc = 0 [ 5648.918745] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2d:0x0]: rc = 0 [ 5648.933596] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2e:0x0]: rc = 0 [ 5648.942996] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2f:0x0]: rc = 0 [ 5648.957937] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x1:0x0]: rc = 0 [ 5648.972649] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x2:0x0]: rc = 0 [ 5648.981145] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x3:0x0]: rc = 0 [ 5648.989192] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20001:0x0]: rc = 0 [ 5648.998794] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20002:0x0]: rc = 0 [ 5649.017951] Lustre: 151969:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20003:0x0]: rc = 0 [ 5649.501443] Lustre: server umount lustre-MDT0001 complete [ 5662.094648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5662.254519] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace' with [0x200000003:0x13:0x0]: rc = 0 [ 5662.263725] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5662.276107] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0x9:0x0]: rc = 0 [ 5662.298250] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000003:0xc:0x0]: rc = 0 [ 5662.310987] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000-MDT0000' with [0x200000005:0x20003:0x0]: rc = 0 [ 5662.333447] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5662.349810] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xa:0x0]: rc = 0 [ 5662.374464] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5662.399983] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5662.409845] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000-MDT0000' with [0x200000005:0x20001:0x0]: rc = 0 [ 5662.430846] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000003:0xd:0x0]: rc = 0 [ 5662.450675] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000003:0xe:0x0]: rc = 0 [ 5662.470527] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xb:0x0]: rc = 0 [ 5662.485824] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000-MDT0000' with [0x200000005:0x20002:0x0]: rc = 0 [ 5662.508803] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_08' with [0x200000003:0x2d:0x0]: rc = 0 [ 5662.542859] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_10' with [0x200000003:0x1e:0x0]: rc = 0 [ 5662.549564] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_01' with [0x200000003:0x15:0x0]: rc = 0 [ 5662.555977] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_03' with [0x200000003:0x17:0x0]: rc = 0 [ 5662.565803] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_10' with [0x200000003:0x2f:0x0]: rc = 0 [ 5662.572993] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_06' with [0x200000003:0x2b:0x0]: rc = 0 [ 5662.580334] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_09' with [0x200000003:0x1d:0x0]: rc = 0 [ 5662.589473] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_04' with [0x200000003:0x29:0x0]: rc = 0 [ 5662.615341] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_15' with [0x200000003:0x34:0x0]: rc = 0 [ 5662.636960] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_14' with [0x200000003:0x22:0x0]: rc = 0 [ 5662.653853] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_11' with [0x200000003:0x30:0x0]: rc = 0 [ 5662.668242] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_12' with [0x200000003:0x20:0x0]: rc = 0 [ 5662.684467] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_00' with [0x200000003:0x25:0x0]: rc = 0 [ 5662.715807] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_07' with [0x200000003:0x2c:0x0]: rc = 0 [ 5662.731565] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_02' with [0x200000003:0x16:0x0]: rc = 0 [ 5662.747695] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_01' with [0x200000003:0x26:0x0]: rc = 0 [ 5662.777702] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_14' with [0x200000003:0x33:0x0]: rc = 0 [ 5662.809686] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_06' with [0x200000003:0x1a:0x0]: rc = 0 [ 5662.860798] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_13' with [0x200000003:0x21:0x0]: rc = 0 [ 5662.876552] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_13' with [0x200000003:0x32:0x0]: rc = 0 [ 5662.884790] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_15' with [0x200000003:0x23:0x0]: rc = 0 [ 5662.892684] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_05' with [0x200000003:0x2a:0x0]: rc = 0 [ 5662.900497] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_08' with [0x200000003:0x1c:0x0]: rc = 0 [ 5662.911198] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_07' with [0x200000003:0x1b:0x0]: rc = 0 [ 5662.924776] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_05' with [0x200000003:0x19:0x0]: rc = 0 [ 5662.938035] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_03' with [0x200000003:0x28:0x0]: rc = 0 [ 5662.946786] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_02' with [0x200000003:0x27:0x0]: rc = 0 [ 5662.955550] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_04' with [0x200000003:0x18:0x0]: rc = 0 [ 5662.963316] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_09' with [0x200000003:0x2e:0x0]: rc = 0 [ 5662.970524] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_00' with [0x200000003:0x14:0x0]: rc = 0 [ 5662.980327] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_12' with [0x200000003:0x31:0x0]: rc = 0 [ 5662.990887] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_11' with [0x200000003:0x1f:0x0]: rc = 0 [ 5663.000903] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x2:0x0]: rc = 0 [ 5663.015814] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 5663.028673] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 5663.039624] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 5663.050761] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 5663.061699] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 5663.072348] Lustre: 152467:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 5679.496677] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5691.070792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5691.245285] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5691.259857] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace' with [0x200000003:0xd:0x0]: rc = 0 [ 5691.273686] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x5:0x0]: rc = 0 [ 5691.285715] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5691.299651] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x20000' with [0x200000003:0x6:0x0]: rc = 0 [ 5691.313675] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2020000-MDT0001' with [0x200000005:0x20003:0x0]: rc = 0 [ 5691.330854] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2020000' with [0x200000003:0x8:0x0]: rc = 0 [ 5691.341293] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x4:0x0]: rc = 0 [ 5691.359640] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5691.373623] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x3:0x0]: rc = 0 [ 5691.382645] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1020000-MDT0001' with [0x200000005:0x20002:0x0]: rc = 0 [ 5691.391792] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x20000-MDT0001' with [0x200000005:0x20001:0x0]: rc = 0 [ 5691.399460] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5691.406495] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5691.440072] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_01' with [0x200000003:0xf:0x0]: rc = 0 [ 5691.454696] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_14' with [0x200000003:0x1c:0x0]: rc = 0 [ 5691.468482] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_11' with [0x200000003:0x19:0x0]: rc = 0 [ 5691.502466] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_04' with [0x200000003:0x23:0x0]: rc = 0 [ 5691.524512] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_06' with [0x200000003:0x14:0x0]: rc = 0 [ 5691.558841] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_07' with [0x200000003:0x26:0x0]: rc = 0 [ 5691.577273] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_15' with [0x200000003:0x1d:0x0]: rc = 0 [ 5691.593294] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_08' with [0x200000003:0x16:0x0]: rc = 0 [ 5691.609713] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_13' with [0x200000003:0x2c:0x0]: rc = 0 [ 5691.626510] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_02' with [0x200000003:0x21:0x0]: rc = 0 [ 5691.648147] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_12' with [0x200000003:0x2b:0x0]: rc = 0 [ 5691.674240] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_13' with [0x200000003:0x1b:0x0]: rc = 0 [ 5691.688274] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_15' with [0x200000003:0x2e:0x0]: rc = 0 [ 5691.702772] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_03' with [0x200000003:0x22:0x0]: rc = 0 [ 5691.720436] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_01' with [0x200000003:0x20:0x0]: rc = 0 [ 5691.735400] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_10' with [0x200000003:0x18:0x0]: rc = 0 [ 5691.747449] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_14' with [0x200000003:0x2d:0x0]: rc = 0 [ 5691.758326] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_10' with [0x200000003:0x29:0x0]: rc = 0 [ 5691.767370] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_00' with [0x200000003:0xe:0x0]: rc = 0 [ 5691.775719] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_09' with [0x200000003:0x17:0x0]: rc = 0 [ 5691.783178] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_09' with [0x200000003:0x28:0x0]: rc = 0 [ 5691.799515] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_05' with [0x200000003:0x24:0x0]: rc = 0 [ 5691.810850] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_00' with [0x200000003:0x1f:0x0]: rc = 0 [ 5691.836244] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_08' with [0x200000003:0x27:0x0]: rc = 0 [ 5691.858296] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_07' with [0x200000003:0x15:0x0]: rc = 0 [ 5691.878499] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_12' with [0x200000003:0x1a:0x0]: rc = 0 [ 5691.890479] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_05' with [0x200000003:0x13:0x0]: rc = 0 [ 5691.911368] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_06' with [0x200000003:0x25:0x0]: rc = 0 [ 5691.934920] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_03' with [0x200000003:0x11:0x0]: rc = 0 [ 5691.949628] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_04' with [0x200000003:0x12:0x0]: rc = 0 [ 5691.970809] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_02' with [0x200000003:0x10:0x0]: rc = 0 [ 5691.987961] Lustre: 153215:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_11' with [0x200000003:0x2a:0x0]: rc = 0 [ 5692.324305] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:106 to 0x2c0000400:129) [ 5692.373333] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:105 to 0x280000400:129) [ 5697.599247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:129) [ 5697.600356] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 5697.924575] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5712.217255] Lustre: DEBUG MARKER: == sanity-scrub test 17a: ENOSPC on OI insert shouldn't leak inodes ========================================================== 16:32:08 (1787862728) [ 5713.495659] Lustre: *** cfs_fail_loc=19d, val=0*** [ 5713.498807] Lustre: Skipped 123 previous similar messages [ 5715.959457] Lustre: Failing over lustre-MDT0000 [ 5716.355189] Lustre: server umount lustre-MDT0000 complete [ 5717.987249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.011836] Lustre: Skipped 20 previous similar messages [ 5734.367379] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5734.376904] LustreError: Skipped 2 previous similar messages [ 5738.975826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5744.638213] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xebca28950a483eac [ 5744.648169] Lustre: MGC192.168.202.138@tcp: Connection restored to 0@lo (at 0@lo) [ 5744.661275] Lustre: Skipped 11 previous similar messages [ 5744.873371] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5744.880138] Lustre: Skipped 2 previous similar messages [ 5746.666625] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5746.674846] Lustre: Skipped 2 previous similar messages [ 5750.330271] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5750.338445] Lustre: Skipped 2 previous similar messages [ 5750.404123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:161) [ 5750.404524] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:161) [ 5750.976844] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5755.378868] Lustre: DEBUG MARKER: == sanity-scrub test 17b: ENOSPC on .. insertion shouldn't leak inodes ========================================================== 16:32:52 (1787862772) [ 5756.792089] Lustre: *** cfs_fail_loc=19e, val=0*** [ 5759.553785] Lustre: Failing over lustre-MDT0000 [ 5760.366367] Lustre: server umount lustre-MDT0000 complete [ 5776.863500] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862778/real 1787862778] req@ffff946d15d08b40 x1874704391026560/t0(0) o400->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787862794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5776.910179] Lustre: 16400:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [ 5781.726629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.106751] LustreError: 16398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff946d375521c0 x1874704391035136/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5792.829124] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:193) [ 5792.833589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:193) [ 5794.097604] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5798.559799] Lustre: DEBUG MARKER: == sanity-scrub test 18: test mount -o resetoi to recreate OI files ========================================================== 16:33:34 (1787862814) [ 5824.942604] Lustre: Failing over lustre-MDT0000 [ 5827.199565] Lustre: server umount lustre-MDT0000 complete [ 5831.828282] LustreError: 139496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862850 with bad export cookie 16990437164796363483 [ 5831.829179] Lustre: Failing over lustre-MDT0001 [ 5831.836359] LustreError: 139496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 12 previous similar messages [ 5832.245670] Lustre: server umount lustre-MDT0001 complete [ 5841.681906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5862.146134] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5871.454476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5871.667164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 5871.675105] LustreError: Skipped 4 previous similar messages [ 5871.756454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5871.762483] Lustre: Skipped 10 previous similar messages [ 5871.804552] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 5871.806158] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:193) [ 5877.143536] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5877.328306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:234 to 0x280000401:257) [ 5877.331637] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:257) [ 5885.093035] Lustre: Failing over lustre-MDT0000 [ 5885.465052] Lustre: server umount lustre-MDT0000 complete [ 5890.509620] Lustre: Failing over lustre-MDT0001 [ 5891.190947] Lustre: server umount lustre-MDT0001 complete [ 5902.639705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5902.680500] Lustre: lustre-MDT0000: reset Object Index mappings [ 5902.693143] Lustre: Skipped 1 previous similar message [ 5915.110252] LustreError: 16398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff946d328ca940 x1874704391171968/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5921.737934] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5932.987884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5933.333474] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:225) [ 5933.337945] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:225) [ 5938.679431] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5938.822365] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:289) [ 5938.826727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:234 to 0x280000401:289) [ 5952.757213] Lustre: DEBUG MARKER: == sanity-scrub test 19: LFSCK can fix multiple linked files on OST ========================================================== 16:36:08 (1787862968) [ 5964.258355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5964.281896] Lustre: Skipped 4 previous similar messages [ 5969.378080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5969.380464] Lustre: Skipped 2 previous similar messages [ 5970.404658] Lustre: server umount lustre-MDT0000 complete [ 5978.098193] Lustre: server umount lustre-MDT0001 complete [ 5994.046544] Lustre: server umount lustre-OST0000 complete [ 6000.323350] Lustre: server umount lustre-OST0001 complete [ 6009.396797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6022.945834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 6038.943790] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6044.063323] LustreError: 161996:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 4: rc = -110 [ 6079.062511] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6085.860478] Lustre: Failing over lustre-OST0000 [ 6086.059675] Lustre: server umount lustre-OST0000 complete [ 6094.155981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6106.817893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 6122.591546] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6127.711343] LustreError: 163524:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 4: rc = -110 [ 6162.268747] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6172.856813] Lustre: DEBUG MARKER: == sanity-scrub test 20: Don't trigger OI scrub for irreparable oi repeatedly ========================================================== 16:39:49 (1787863189) [ 6190.743894] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6203.826842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6204.443066] LustreError: 163549:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.467755] LustreError: 163549:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 127 previous similar messages [ 6204.679962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:385) [ 6210.233227] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6221.172569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6221.520172] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:257) [ 6227.065995] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6230.776581] Lustre: 166435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6248.594988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6254.079194] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:257) [ 6254.085865] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:321) [ 6257.482180] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6268.127792] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6276.277548] Lustre: *** cfs_fail_loc=193, val=0*** [ 6278.594053] Lustre: Failing over lustre-MDT0000 [ 6278.721758] Lustre: *** cfs_fail_loc=193, val=0*** [ 6279.009544] Lustre: server umount lustre-MDT0000 complete [ 6289.432562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 6289.826908] Lustre: *** cfs_fail_loc=193, val=0*** [ 6295.074638] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:353) [ 6295.075891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:417) [ 6296.108598] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6300.144649] Lustre: *** cfs_fail_loc=193, val=0*** [ 6300.157218] Lustre: Skipped 3 previous similar messages [ 6300.759698] Lustre: *** cfs_fail_loc=19f, val=0*** [ 6300.760640] Lustre: lustre-MDT0000: trigger OI scrub by RPC for [0x200003ab1:0x2:0x0]/16843009 with flags 0x4a: rc = 0 [ 6300.762940] Lustre: Skipped 52 previous similar messages [ 6318.792891] Lustre: DEBUG MARKER: == sanity-scrub test 21: don't hang MDS recovery when failed to get update log ========================================================== 16:42:14 (1787863334) [ 6322.950628] Lustre: Failing over lustre-MDT0000 [ 6322.994994] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 6323.368555] Lustre: server umount lustre-MDT0000 complete [ 6325.729790] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6325.760325] Lustre: Skipped 29 previous similar messages [ 6327.991610] Lustre: Failing over lustre-MDT0001 [ 6328.116176] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 6328.352495] Lustre: server umount lustre-MDT0001 complete [ 6331.671423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6340.050910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6353.376124] LustreError: 16398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff946d3b7ccf00 x1874704391351680/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6354.479903] Lustre: lustre-MDT0000: trigger OI scrub by RPC for [0x200000400:0x1:0x0]/30 with flags 0x4a: rc = 0 [ 6354.500906] Lustre: 170217:0:(lod_sub_object.c:941:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't open llog [0x200000400:0x1:0x0]: rc = -115 [ 6354.522760] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6354.529298] LustreError: 170217:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0000-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6354.569648] Lustre: Skipped 4 previous similar messages [ 6361.160825] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6366.692312] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6366.704774] Lustre: Skipped 22 previous similar messages [ 6370.788565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.481465] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 6372.495132] Lustre: Skipped 4 previous similar messages [ 6376.498222] Lustre: lustre-MDT0000: trigger OI scrub by RPC for [0x200000401:0x1:0x0]/32 with flags 0x4a: rc = 0 [ 6377.644664] LustreError: 170942:0:(update_trans.c:1081:top_trans_stop()) lustre-MDT0000-osp-MDT0001: stop trans failed: rc = -20 [ 6377.673595] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6377.688090] Lustre: Skipped 4 previous similar messages [ 6377.758961] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:385) [ 6377.766689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:449) [ 6377.830359] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:289) [ 6377.830945] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:289) [ 6377.959113] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6385.248185] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 200 [ 6392.741645] Lustre: DEBUG MARKER: == sanity-scrub test 22: LFSCK can recreate or fix the LASTID on MDT/OST ========================================================== 16:43:29 (1787863409) [ 6402.021574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6402.034071] Lustre: Skipped 3 previous similar messages [ 6405.642614] Lustre: server umount lustre-MDT0000 complete [ 6409.745062] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6409.755700] LustreError: Skipped 6 previous similar messages [ 6410.144691] Lustre: server umount lustre-MDT0001 complete [ 6424.470682] Lustre: server umount lustre-OST0000 complete [ 6438.858797] Lustre: server umount lustre-OST0001 complete [ 6445.978933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6458.096554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.253340] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6468.296964] Lustre: Failing over lustre-MDT0000 [ 6468.324154] LustreError: 173338:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6468.330491] LustreError: 173338:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6468.337347] LustreError: 173338:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 6468.817519] Lustre: server umount lustre-MDT0000 complete [ 6475.118105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6489.016525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 6489.607941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6489.612586] Lustre: Skipped 11 previous similar messages [ 6494.921548] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6504.410956] Lustre: DEBUG MARKER: === sanity-scrub: start setup 16:45:21 (1787863521) === [ 6506.725552] LustreError: 174974:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6506.744748] LustreError: 174974:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6506.751512] LustreError: 174974:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 6507.153046] Lustre: server umount lustre-MDT0000 complete [ 6548.765549] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 6561.242650] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6617.404782] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6631.810805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6632.010834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6632.033875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6632.138111] Lustre: lustre-MDT0000: new disk, initializing [ 6632.240952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6637.615381] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6651.642377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6651.743113] Lustre: 180338:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6651.759167] Lustre: 180338:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 6651.808060] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6651.812722] Lustre: Skipped 1 previous similar message [ 6651.926672] Lustre: lustre-MDT0001: new disk, initializing [ 6652.033056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6652.055479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6657.064768] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6662.276832] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6673.092246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6673.428294] Lustre: lustre-OST0000: new disk, initializing [ 6673.434562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6673.447885] Lustre: 182276:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6674.902405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6674.927073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6675.057670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6680.671664] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6695.860271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6696.114682] Lustre: lustre-OST0001: new disk, initializing [ 6696.119184] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6696.130372] Lustre: 183302:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6698.054314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6698.061650] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6698.160753] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6703.481254] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6714.355950] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6718.666426] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6728.546072] Lustre: DEBUG MARKER: === sanity-scrub: finish setup 16:49:04 (1787863744) === [ 6731.262821] Lustre: DEBUG MARKER: == sanity-scrub test complete, duration 6474 sec ========= 16:49:07 (1787863747) [ 6734.419178] Lustre: DEBUG MARKER: === sanity-scrub: start cleanup 16:49:10 (1787863750) === [ 6739.062576] Lustre: DEBUG MARKER: === sanity-scrub: finish cleanup 16:49:15 (1787863755) === [ 6747.978416] Lustre: server umount lustre-MDT0000 complete [ 6749.174424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6749.189872] LustreError: Skipped 5 previous similar messages [ 6759.248555] LustreError: 180330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863777 with bad export cookie 16990437164796418923 [ 6759.252613] LustreError: 180330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 11 previous similar messages [ 6759.481870] Lustre: server umount lustre-MDT0001 complete [ 6776.287298] Lustre: 16402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863778/real 1787863778] req@ffff946d3b7ced00 x1874704391482368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787863794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6776.318787] Lustre: 16402:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 37 previous similar messages [ 6779.529214] Lustre: server umount lustre-OST0000 complete [ 6789.590890] Lustre: server umount lustre-OST0001 complete [ 6809.496861] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 6813.981929] Key type lgssc unregistered [ 6814.427854] LNet: 186720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6814.447329] LNetError: 186720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6815.527494] LNet: Removed LNI 192.168.202.138@tcp [ 6817.482152] Key type .llcrypt unregistered [ 6817.483653] Key type ._llcrypt unregistered