[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483210224 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002380] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007398] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010099] LSM: Security Framework initializing [ 0.011033] Yama: becoming mindful. [ 0.012022] SELinux: Initializing. [ 0.013047] *** VALIDATE selinux *** [ 0.020293] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024846] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027100] *** VALIDATE tmpfs *** [ 0.029249] *** VALIDATE proc *** [ 0.030108] *** VALIDATE cgroup *** [ 0.031000] *** VALIDATE cgroup2 *** [ 0.031278] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034031] Spectre V2 : User space: Vulnerable [ 0.035010] Speculative Store Bypass: Vulnerable [ 0.037701] debug: unmapping init [mem 0xffffffff9ec59000-0xffffffff9ec60fff] [ 0.040228] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041692] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042023] ... version: 2 [ 0.043012] ... bit width: 48 [ 0.044008] ... generic registers: 4 [ 0.045011] ... value mask: 0000ffffffffffff [ 0.046012] ... max period: 00007fffffffffff [ 0.047015] ... fixed-purpose events: 3 [ 0.048010] ... event mask: 000000070000000f [ 0.049300] rcu: Hierarchical SRCU implementation. [ 0.051544] smp: Bringing up secondary CPUs ... [ 0.052600] x86: Booting SMP configuration: [ 0.053024] .... node #0, CPUs: #1 #2 #3 [ 0.056321] smp: Brought up 1 node, 4 CPUs [ 0.058012] smpboot: Max logical packages: 1 [ 0.059017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127023] node 0 deferred pages initialised in 66ms [ 0.131009] devtmpfs: initialized [ 0.132257] x86/mm: Memory block size: 128MB [ 0.134929] gcov: version magic: 0x41383552 [ 0.138341] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144280] pinctrl core: initialized pinctrl subsystem [ 0.146189] [ 0.146768] ************************************************************* [ 0.149015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151014] ** ** [ 0.153012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155011] ** ** [ 0.157010] ** This means that this kernel is built to expose internal ** [ 0.158009] ** IOMMU data structures, which may compromise security on ** [ 0.160008] ** your system. ** [ 0.161011] ** ** [ 0.162009] ** If you see this message and you are not debugging the ** [ 0.165014] ** kernel, report this immediately to your vendor! ** [ 0.167013] ** ** [ 0.169008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171008] ************************************************************* [ 0.172611] NET: Registered protocol family 16 [ 0.174337] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180085] cpuidle: using governor menu [ 0.181484] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183437] PCI: Using configuration type 1 for base access [ 0.184101] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.191029] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195072] cryptd: max_cpu_qlen set to 1000 [ 0.198066] ACPI: Added _OSI(Module Device) [ 0.200015] ACPI: Added _OSI(Processor Device) [ 0.201013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.203014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208719] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215369] ACPI: Interpreter enabled [ 0.217058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218013] ACPI: Using IOAPIC for interrupt routing [ 0.219127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222564] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.230126] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.234018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.240084] acpiphp: Slot [2] registered [ 0.241080] acpiphp: Slot [5] registered [ 0.242065] acpiphp: Slot [6] registered [ 0.243025] acpiphp: Slot [7] registered [ 0.244019] acpiphp: Slot [8] registered [ 0.245047] acpiphp: Slot [9] registered [ 0.246074] acpiphp: Slot [10] registered [ 0.247109] acpiphp: Slot [3] registered [ 0.248061] acpiphp: Slot [4] registered [ 0.249059] acpiphp: Slot [11] registered [ 0.250054] acpiphp: Slot [12] registered [ 0.251081] acpiphp: Slot [13] registered [ 0.252065] acpiphp: Slot [14] registered [ 0.253094] acpiphp: Slot [15] registered [ 0.254060] acpiphp: Slot [16] registered [ 0.255036] acpiphp: Slot [17] registered [ 0.255961] acpiphp: Slot [18] registered [ 0.257067] acpiphp: Slot [19] registered [ 0.258036] acpiphp: Slot [20] registered [ 0.258973] acpiphp: Slot [21] registered [ 0.260062] acpiphp: Slot [22] registered [ 0.261007] acpiphp: Slot [23] registered [ 0.262026] acpiphp: Slot [24] registered [ 0.263009] acpiphp: Slot [25] registered [ 0.263922] acpiphp: Slot [26] registered [ 0.265087] acpiphp: Slot [27] registered [ 0.267132] acpiphp: Slot [28] registered [ 0.268085] acpiphp: Slot [29] registered [ 0.269072] acpiphp: Slot [30] registered [ 0.270073] acpiphp: Slot [31] registered [ 0.271053] PCI host bridge to bus 0000:00 [ 0.271967] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.274014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.275013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.277016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.279015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.281017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.282183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.284993] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.287175] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.294016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.302465] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.304016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.306018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.308018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.311463] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.313586] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.315036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.317699] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.322011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.332050] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.336013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.339225] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.346013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.354017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.379017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.393047] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.404016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.413015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.434019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.446228] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.455012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.463026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.492018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.505639] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.514015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.521019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.538021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.548792] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.556014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.563019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.596028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.611000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.618018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.624029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.638028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.647803] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.651368] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.653302] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.654384] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.658256] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.664271] iommu: Default domain type: Passthrough [ 0.666491] SCSI subsystem initialized [ 0.668124] ACPI: bus type USB registered [ 0.669120] usbcore: registered new interface driver usbfs [ 0.671083] usbcore: registered new interface driver hub [ 0.673082] usbcore: registered new device driver usb [ 0.675191] pps_core: LinuxPPS API ver. 1 registered [ 0.677012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.680051] PTP clock support registered [ 0.681122] EDAC MC: Ver: 3.0.0 [ 0.683243] PCI: Using ACPI for IRQ routing [ 0.685034] NetLabel: Initializing [ 0.686011] NetLabel: domain hash size = 128 [ 0.687010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.689073] NetLabel: unlabeled traffic allowed by default [ 0.691139] vgaarb: loaded [ 0.692246] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.694013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.699335] clocksource: Switched to clocksource kvm-clock [ 0.810838] VFS: Disk quotas dquot_6.6.0 [ 0.812045] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.813454] *** VALIDATE ramfs *** [ 0.814432] *** VALIDATE hugetlbfs *** [ 0.815822] pnp: PnP ACPI init [ 0.817882] pnp: PnP ACPI: found 6 devices [ 0.833311] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.836043] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.837581] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.839017] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.840646] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.842133] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.843963] NET: Registered protocol family 2 [ 0.846047] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.850368] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.853575] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.858707] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.862136] TCP: Hash tables configured (established 65536 bind 65536) [ 0.864799] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.867589] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.870341] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.873362] NET: Registered protocol family 1 [ 0.876094] RPC: Registered named UNIX socket transport module. [ 0.878012] RPC: Registered udp transport module. [ 0.879090] RPC: Registered tcp transport module. [ 0.880536] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.882819] NET: Registered protocol family 44 [ 0.884102] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.885887] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.887258] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.889108] PCI: CLS 0 bytes, default 64 [ 0.890486] Unpacking initramfs... [ 2.248788] debug: unmapping init [mem 0xffff9fc0fcc54000-0xffff9fc0fffbffff] [ 2.251463] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.253787] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.256285] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.761024] Initialise system trusted keyrings [ 2.762239] Key type blacklist registered [ 2.763772] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.777515] zbud: loaded [ 2.782518] *** VALIDATE nfs *** [ 2.783678] *** VALIDATE nfs4 *** [ 2.786453] pstore: using deflate compression [ 2.790938] Platform Keyring initialized [ 2.908552] NET: Registered protocol family 38 [ 2.910663] Key type asymmetric registered [ 2.912456] Asymmetric key parser 'x509' registered [ 2.914176] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.916951] io scheduler mq-deadline registered [ 2.920175] io scheduler kyber registered [ 2.921936] io scheduler bfq registered [ 2.924179] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.926709] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.929818] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.932979] ACPI: Power Button [PWRF] [ 2.939466] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.948195] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.966193] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.974948] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.992974] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.021251] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.048182] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.052889] Non-volatile memory driver v1.3 [ 3.054724] Linux agpgart interface v0.103 [ 3.082680] virtio_blk virtio1: [vda] 149944 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.085379] vda: detected capacity change from 0 to 76771328 [ 3.101613] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.104896] vdb: detected capacity change from 0 to 1073741824 [ 3.119439] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.122730] vdc: detected capacity change from 0 to 2621440000 [ 3.136883] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.139985] vdd: detected capacity change from 0 to 2621440000 [ 3.153921] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.156712] vde: detected capacity change from 0 to 4294967296 [ 3.169280] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.172088] vdf: detected capacity change from 0 to 4294967296 [ 3.179222] libphy: Fixed MDIO Bus: probed [ 3.189918] usbcore: registered new interface driver usbserial_generic [ 3.192162] usbserial: USB Serial support registered for generic [ 3.194421] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.199198] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.201264] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.204033] mousedev: PS/2 mouse device common for all mice [ 3.206713] rtc_cmos 00:05: RTC can wake from S4 [ 3.209289] rtc_cmos 00:05: registered as rtc0 [ 3.214336] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.216742] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.222219] intel_pstate: CPU model not supported [ 3.223111] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.232404] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.232467] hid: raw HID events driver (C) Jiri Kosina [ 3.238611] usbcore: registered new interface driver usbhid [ 3.240187] usbhid: USB HID core driver [ 3.241838] drop_monitor: Initializing network drop monitor service [ 3.245901] Initializing XFRM netlink socket [ 3.248255] NET: Registered protocol family 10 [ 3.251730] Segment Routing with IPv6 [ 3.253397] NET: Registered protocol family 17 [ 3.255936] mpls_gso: MPLS GSO support [ 3.262471] RAS: Correctable Errors collector initialized. [ 3.264380] AVX version of gcm_enc/dec engaged. [ 3.265878] AES CTR mode by8 optimization enabled [ 3.353949] sched_clock: Marking stable (3353919714, 0)->(4177474126, -823554412) [ 3.357811] registered taskstats version 1 [ 3.360535] Loading compiled-in X.509 certificates [ 3.363120] zswap: loaded using pool lzo/zbud [ 3.388419] Key type big_key registered [ 3.402562] Key type encrypted registered [ 3.404738] ima: No TPM chip found, activating TPM-bypass! [ 3.407458] ima: Allocated hash algorithm: sha1 [ 3.409847] ima: No architecture policies found [ 3.411836] evm: Initialising EVM extended attributes: [ 3.413039] evm: security.selinux [ 3.414087] evm: security.ima [ 3.414955] evm: security.capability [ 3.416122] evm: HMAC attrs: 0x1 [ 3.418287] rtc_cmos 00:05: setting system clock to 2026-09-13 12:09:37 UTC (1789301377) [ 3.424720] debug: unmapping init [mem 0xffffffff9fc03000-0xffffffff9fdfffff] [ 3.427663] debug: unmapping init [mem 0xffffffff9e982000-0xffffffff9ec58fff] [ 3.436150] Write protecting the kernel read-only data: 28672k [ 3.439211] debug: unmapping init [mem 0xffffffff9d003000-0xffffffff9d1fffff] [ 3.442587] debug: unmapping init [mem 0xffffffff9d914000-0xffffffff9d9fffff] [ 3.491653] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.500731] systemd[1]: Detected virtualization kvm. [ 3.504114] systemd[1]: Detected architecture x86-64. [ 3.506262] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.532048] systemd[1]: No hostname configured. [ 3.533718] systemd[1]: Set hostname to . [ 3.535360] random: systemd: uninitialized urandom read (16 bytes read) [ 3.537979] systemd[1]: Initializing machine ID from random generator. [ 3.586718] random: ln: uninitialized urandom read (6 bytes read) [ 3.692219] random: systemd: uninitialized urandom read (16 bytes read) [ 3.695108] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.702133] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.706027] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.368529] device-mapper: uevent: version 1.0.3 [ 4.370700] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.653499] virtio_net virtio0 ens2: renamed from eth0 [ 5.722620] random: fast init done [ 5.976322] scsi host0: ata_piix [ 5.987053] scsi host1: ata_piix [ 5.995341] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.008233] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.934700] random: crng init done [ 10.937089] random: 7 urandom warning(s) missed due to ratelimiting [ 12.413922] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.957963] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.959406] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.744857] SELinux: Disabled at runtime. [ 17.863117] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.896055] systemd[1]: Detected virtualization kvm. [ 17.897850] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.215596] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.218096] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.222681] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.225285] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.229220] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.233943] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.238934] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 19.461955] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 20.535635] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.470950] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.632620] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.409619] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.532901] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 26.774485] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 27.609870] NFS: Registering the id_resolver key type [ 27.620613] Key type id_resolver registered [ 27.622182] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg259-server login: [ 85.229513] libcfs: loading out-of-tree module taints kernel. [ 85.346331] Key type ._llcrypt registered [ 85.351401] Key type .llcrypt registered [ 85.569329] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 103.768432] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 105.952755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.974338] alg: No test for adler32 (adler32-zlib) [ 107.389123] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 108.342125] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 110.216706] Key type lgssc registered [ 112.008232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.677994] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 168.385475] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 182.100185] hrtimer: interrupt took 1965993 ns [ 183.009648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 183.051094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.437331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.506924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 184.716390] Lustre: lustre-MDT0000: new disk, initializing [ 184.882400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.909664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 190.238368] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.098465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.190114] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 204.243519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 204.248084] Lustre: Skipped 1 previous similar message [ 204.421423] Lustre: lustre-MDT0001: new disk, initializing [ 204.498632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.537463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 204.559329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.480884] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.858587] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.186614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.395768] Lustre: lustre-OST0000: new disk, initializing [ 224.398510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.403744] Lustre: 8458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.480188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.536438] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.919628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.929302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.074422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 244.468422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.625838] Lustre: lustre-OST0001: new disk, initializing [ 244.628772] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 244.638883] Lustre: 9531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.720192] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.324355] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.933987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 251.947262] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 252.027526] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 263.014918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 273.967814] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 280.910671] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing check_logdir /tmp/testlogs/ [ 285.277098] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing yml_node [ 289.830187] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 291.817191] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 294.055333] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 295.514559] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-scrub ============----- Sun Sep 13 08:14:29 EDT 2026 [ 310.405928] Lustre: DEBUG MARKER: excepting tests: [ 319.157826] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 331.746622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.751455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.768373] Lustre: Skipped 1 previous similar message [ 331.795841] Lustre: Skipped 3 previous similar messages [ 334.636364] Lustre: server umount lustre-MDT0000 complete [ 341.999044] LustreError: 6532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.015326] LustreError: 6532:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 345.631133] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789301719 with bad export cookie 16508121503327547012 [ 345.632965] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.672212] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 346.031569] Lustre: server umount lustre-MDT0001 complete [ 363.488283] Lustre: 3653:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789301721/real 1789301721] req@ffff9fc17716a300 x1876218590893696/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789301737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 363.524883] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.541707] Lustre: Skipped 2 previous similar messages [ 365.673891] Lustre: server umount lustre-OST0000 complete [ 366.560148] Lustre: 3652:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789301724/real 1789301724] req@ffff9fc042e06d80 x1876218590893952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789301740 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 368.352514] Lustre: 3654:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789301726/real 1789301726] req@ffff9fc177169500 x1876218590894208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789301742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 371.745559] Lustre: 3652:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789301729/real 1789301729] req@ffff9fc043349880 x1876218590894592/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789301745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 374.972707] Lustre: server umount lustre-OST0001 complete [ 391.324575] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 395.624427] Key type lgssc unregistered [ 396.052274] LNet: 14803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.066281] LNetError: 14803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 396.093390] LNet: Removed LNI 192.168.202.159@tcp [ 397.531589] Key type .llcrypt unregistered [ 397.536672] Key type ._llcrypt unregistered [ 425.542344] Key type ._llcrypt registered [ 425.544933] Key type .llcrypt registered [ 425.722525] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 443.300051] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 445.088035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 445.207761] alg: No test for adler32 (adler32-zlib) [ 446.287804] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 446.523733] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 448.228149] Key type lgssc registered [ 449.301465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 500.894075] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 513.749630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 513.786696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.104756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 515.157652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 515.301984] Lustre: lustre-MDT0000: new disk, initializing [ 515.367660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.378642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 520.125307] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 532.337814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.468648] Lustre: 19254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 532.510524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 532.514466] Lustre: Skipped 1 previous similar message [ 532.622704] Lustre: lustre-MDT0001: new disk, initializing [ 532.708068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 532.738665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 532.754632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 537.280233] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 542.670907] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 551.694072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 552.100885] Lustre: lustre-OST0000: new disk, initializing [ 552.104957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 552.109820] Lustre: 21192:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 552.178923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 557.600984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 557.614458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 557.697756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 558.825693] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 572.331737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 572.474611] Lustre: lustre-OST0001: new disk, initializing [ 572.478644] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 572.485793] Lustre: 22217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 572.564590] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 579.164499] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 581.136586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 581.152152] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 581.220051] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 589.268627] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 596.982627] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 605.320378] Lustre: DEBUG MARKER: === sanity-scrub: finish setup 08:19:38 (1789301978) === [ 606.400759] Lustre: DEBUG MARKER: == sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 08:19:40 (1789301980) [ 626.295495] Lustre: Failing over lustre-MDT0000 [ 626.531218] Lustre: server umount lustre-MDT0000 complete [ 627.188962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.206345] Lustre: Skipped 3 previous similar messages [ 630.277088] LustreError: 19247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302004 with bad export cookie 10810197469120703773 [ 630.278598] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.283824] LustreError: 19247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 630.284255] Lustre: Failing over lustre-MDT0001 [ 630.676637] Lustre: server umount lustre-MDT0001 complete [ 638.833074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 640.032167] LustreError: 24219:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 640.045484] LustreError: 24219:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9fc179c89180 x1876218945333888/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789302014 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:4294967295 [ 640.076904] LustreError: 24219:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 640.421105] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc043817800 x1876218945335296/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 640.674730] LustreError: 21186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.696075] LustreError: 21186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 640.767422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 640.795692] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 644.988172] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 646.119349] LustreError: 21187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 646.121141] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.204042] LustreError: 24234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.211976] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 647.218782] LustreError: 24234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 648.224348] Lustre: 16416:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302006/real 1789302006] req@ffff9fc177356d80 x1876218945334656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789302022 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 651.296197] Lustre: 16418:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302009/real 1789302009] req@ffff9fc0433f9f80 x1876218945334912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789302025 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 651.339185] Lustre: 16418:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 652.269730] LustreError: 21186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.288767] LustreError: 21186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 653.280872] Lustre: 16416:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302011/real 1789302011] req@ffff9fc177355180 x1876218945335168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789302027 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 653.306661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 653.321756] Lustre: 16416:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 653.602026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 653.681651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 653.715575] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:65) [ 653.719434] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:65) [ 656.672407] Lustre: 16419:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302014/real 1789302014] req@ffff9fc043814000 x1876218945335424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789302030 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 658.355452] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 658.918117] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 658.931687] Lustre: Skipped 1 previous similar message [ 658.978767] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 659.014429] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 659.085996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 659.088132] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:65) [ 671.178780] Lustre: DEBUG MARKER: == sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 08:20:44 (1789302044) [ 692.165767] Lustre: Failing over lustre-MDT0000 [ 694.550278] Lustre: server umount lustre-MDT0000 complete [ 694.760260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 694.768191] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.779641] LustreError: 21187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.849470] Lustre: Skipped 3 previous similar messages [ 698.040404] LustreError: 25747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302072 with bad export cookie 10810197469120720083 [ 698.044506] Lustre: Failing over lustre-MDT0001 [ 698.045611] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 698.049911] LustreError: 25747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 698.564924] Lustre: server umount lustre-MDT0001 complete [ 708.135202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 715.104270] Lustre: 16418:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302073/real 1789302073] req@ffff9fc043925500 x1876218945449472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789302089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 715.149741] Lustre: 16418:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 715.158191] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.166809] Lustre: Skipped 1 previous similar message [ 723.365454] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc042e07b80 x1876218945451904/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 723.775058] LustreError: 21186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 723.800342] LustreError: 21186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 723.896687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 723.931845] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 724.899673] Lustre: 16419:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302082/real 1789302082] req@ffff9fc043815880 x1876218945450240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789302098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 724.925654] Lustre: 16419:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 729.853761] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 734.178919] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 734.188535] Lustre: Skipped 2 previous similar messages [ 739.152418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 739.298039] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 739.351706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 739.501743] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:106 to 0x2c0000400:129) [ 739.519195] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:105 to 0x280000400:129) [ 743.794442] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 744.942621] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 744.954443] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 744.968567] Lustre: Skipped 1 previous similar message [ 745.027037] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 745.119029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 745.120743] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:106 to 0x2c0000401:129) [ 748.982129] Lustre: *** cfs_fail_loc=193, val=0*** [ 755.849367] Lustre: Failing over lustre-MDT0000 [ 756.062927] Lustre: server umount lustre-MDT0000 complete [ 760.300121] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.302685] LustreError: 26622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 760.310298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 760.331737] Lustre: Skipped 4 previous similar messages [ 760.376118] LustreError: 26622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 765.281373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 765.369360] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 765.606535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 765.613542] Lustre: Skipped 1 previous similar message [ 765.654298] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 770.059225] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 771.047771] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 771.048532] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 771.092874] Lustre: Skipped 2 previous similar messages [ 771.129715] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 771.189522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:161) [ 771.190111] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:106 to 0x2c0000401:161) [ 780.358907] Lustre: DEBUG MARKER: == sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 08:22:33 (1789302153) [ 795.480388] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 814.676866] Lustre: 30501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 837.859468] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 841.233975] Lustre: 31637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 853.353746] Lustre: *** cfs_fail_loc=198, val=0*** [ 864.658149] Lustre: Failing over lustre-MDT0000 [ 865.003635] Lustre: server umount lustre-MDT0000 complete [ 868.325946] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.330065] LustreError: 26622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.339332] Lustre: Failing over lustre-MDT0001 [ 868.345104] LustreError: 20204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302242 with bad export cookie 10810197469120748916 [ 868.345667] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 868.348440] Lustre: Skipped 3 previous similar messages [ 868.411796] LustreError: 26622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 868.763050] Lustre: server umount lustre-MDT0001 complete [ 873.222171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 877.837598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 886.373678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 889.633539] Lustre: 16418:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302247/real 1789302247] req@ffff9fc042e05180 x1876218945627008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789302263 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 889.666778] Lustre: 16418:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 889.676874] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 894.181785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 894.224923] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 898.942770] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 908.541250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 908.773357] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 908.779344] Lustre: Skipped 4 previous similar messages [ 908.784887] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 908.791795] Lustre: Skipped 3 previous similar messages [ 908.813955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 908.947815] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:193) [ 908.955124] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:193) [ 912.961929] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 913.909847] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 913.943257] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 913.987194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:202 to 0x280000401:225) [ 913.994207] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:202 to 0x2c0000401:225) [ 925.335570] Lustre: DEBUG MARKER: == sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 08:24:59 (1789302299) [ 945.855286] Lustre: Failing over lustre-MDT0000 [ 946.623336] Lustre: server umount lustre-MDT0000 complete [ 949.736220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 949.737919] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.745474] LustreError: 21186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.745484] LustreError: 21186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 949.787348] Lustre: Skipped 4 previous similar messages [ 951.409904] Lustre: Failing over lustre-MDT0001 [ 951.411304] LustreError: 25747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302325 with bad export cookie 10810197469120776363 [ 951.422598] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 951.456860] LustreError: 25747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 951.925286] Lustre: server umount lustre-MDT0001 complete [ 956.399378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 962.473478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 970.215854] Lustre: 16416:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302328/real 1789302328] req@ffff9fc043816a00 x1876218945744384/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789302344 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 970.257236] Lustre: 16416:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 972.927565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 972.955724] Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64 [ 976.416930] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc17f620000 x1876218945746432/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 976.827753] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 981.791231] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 989.158592] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 989.170715] Lustre: Skipped 4 previous similar messages [ 990.527766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 990.618075] Lustre: lustre-MDT0001: invalid oi count 63, remove them, then set it to 64 [ 990.921531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 991.053522] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 991.131095] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:234 to 0x280000400:257) [ 991.153597] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:233 to 0x2c0000400:257) [ 995.839745] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 996.407708] Lustre: lustre-MDT0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 996.463243] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:266 to 0x2c0000401:289) [ 996.465252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 1016.773042] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1034.200031] Lustre: 38933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1055.020334] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1057.858761] Lustre: 40068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1077.875494] Lustre: Failing over lustre-MDT0000 [ 1078.168914] Lustre: server umount lustre-MDT0000 complete [ 1078.242358] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.247528] LustreError: 35984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.247742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1078.256246] Lustre: Skipped 5 previous similar messages [ 1078.294178] LustreError: 35984:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [ 1081.719632] Lustre: Failing over lustre-MDT0001 [ 1081.720685] LustreError: 25747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302455 with bad export cookie 10810197469120803971 [ 1081.721041] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1081.737107] LustreError: 25747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1081.915845] Lustre: server umount lustre-MDT0001 complete [ 1086.838157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1094.916043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1099.691679] Lustre: 16419:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302457/real 1789302457] req@ffff9fc0433f8000 x1876218945888000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789302473 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1099.717614] Lustre: 16419:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 1106.005990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1106.038903] Lustre: lustre-MDT0000: invalid oi count 58, remove them, then set it to 64 [ 1106.912664] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc175abf800 x1876218945890432/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1107.421234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.429818] Lustre: Skipped 3 previous similar messages [ 1107.483431] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1112.464866] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1118.698590] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1118.704153] Lustre: Skipped 4 previous similar messages [ 1120.677558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1120.730280] Lustre: lustre-MDT0001: invalid oi count 58, remove them, then set it to 64 [ 1121.363073] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:297 to 0x2c0000400:321) [ 1121.363077] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:298 to 0x280000400:321) [ 1125.252445] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1126.371449] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1126.410701] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1126.460355] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:329 to 0x2c0000401:353) [ 1126.462453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:330 to 0x280000401:353) [ 1147.751985] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1165.097705] Lustre: 44786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1185.547410] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1189.773795] Lustre: 45921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1208.919400] Lustre: Failing over lustre-MDT0000 [ 1209.263504] Lustre: server umount lustre-MDT0000 complete [ 1213.409960] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.423384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1213.434762] Lustre: Skipped 4 previous similar messages [ 1213.449301] LustreError: Skipped 1 previous similar message [ 1213.480458] Lustre: Failing over lustre-MDT0001 [ 1213.488134] LustreError: 45940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302587 with bad export cookie 10810197469120831537 [ 1213.488468] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1213.510979] LustreError: 45940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1214.229956] Lustre: server umount lustre-MDT0001 complete [ 1219.417263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1225.607800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1234.913154] Lustre: 16417:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302592/real 1789302592] req@ffff9fc049bb8380 x1876218946033024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789302608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1234.931708] Lustre: 16417:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 1235.925262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1235.943720] Lustre: lustre-MDT0000: invalid oi count 56, remove them, then set it to 64 [ 1238.049404] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc147c46300 x1876218946034688/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1238.403911] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1242.593548] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1250.099460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1250.137929] Lustre: lustre-MDT0001: invalid oi count 56, remove them, then set it to 64 [ 1250.450973] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:361 to 0x2c0000400:385) [ 1250.453360] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:362 to 0x280000400:385) [ 1253.481672] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1253.485494] Lustre: Skipped 4 previous similar messages [ 1253.495389] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1253.518566] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1253.577838] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:394 to 0x2c0000401:417) [ 1253.584096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:393 to 0x280000401:417) [ 1254.607780] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1268.971440] Lustre: DEBUG MARKER: == sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 08:30:42 (1789302642) [ 1281.952253] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1301.323508] Lustre: 50631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1323.626824] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1349.599181] Lustre: Failing over lustre-MDT0000 [ 1350.064764] Lustre: server umount lustre-MDT0000 complete [ 1351.142072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1351.150963] LustreError: 47489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.159036] LustreError: Skipped 1 previous similar message [ 1351.193966] LustreError: 47489:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 23 previous similar messages [ 1353.956486] LustreError: 25747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302728 with bad export cookie 10810197469120859103 [ 1353.961064] Lustre: Failing over lustre-MDT0001 [ 1353.961260] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1353.965888] LustreError: 25747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1354.239662] Lustre: server umount lustre-MDT0001 complete [ 1360.067969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1369.581729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1377.556432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1385.772696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1395.394205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1395.410684] Lustre: lustre-MDT0000: reset Object Index mappings [ 1399.793211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.800641] Lustre: Skipped 3 previous similar messages [ 1399.854879] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1404.639660] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1413.405248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1413.435378] Lustre: lustre-MDT0001: reset Object Index mappings [ 1414.035545] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:426 to 0x2c0000400:449) [ 1414.035707] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:425 to 0x280000400:449) [ 1416.036085] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1416.037280] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1416.047032] Lustre: Skipped 4 previous similar messages [ 1416.120080] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1416.200638] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:458 to 0x2c0000401:481) [ 1416.206461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:457 to 0x280000401:481) [ 1419.528247] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1431.376694] Lustre: DEBUG MARKER: == sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 08:33:24 (1789302804) [ 1454.690685] Lustre: Failing over lustre-MDT0000 [ 1454.853269] Lustre: server umount lustre-MDT0000 complete [ 1458.705894] LustreError: 19247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302832 with bad export cookie 10810197469120886669 [ 1458.710425] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1458.711119] Lustre: Failing over lustre-MDT0001 [ 1458.720065] LustreError: 19247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1459.112339] Lustre: server umount lustre-MDT0001 complete [ 1464.310552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1474.003370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1479.393897] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.409939] Lustre: Skipped 13 previous similar messages [ 1481.994672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1491.160882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1493.984150] Lustre: 16417:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789302852/real 1789302852] req@ffff9fc171aa7480 x1876218946306560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789302868 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1494.001177] Lustre: 16417:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [ 1500.955511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1500.989460] Lustre: lustre-MDT0000: reset Object Index mappings [ 1503.136537] LustreError: 58325:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1503.151056] LustreError: 58325:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9fc171aa6a00 x1876218946307328/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789302877 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1503.170737] LustreError: 58325:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1503.200738] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc1778bd180 x1876218946307968/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1503.719198] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1507.441411] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1515.696444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1515.723050] Lustre: lustre-MDT0001: reset Object Index mappings [ 1516.013819] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1516.061509] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:489 to 0x2c0000400:513) [ 1516.067460] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:490 to 0x280000400:513) [ 1520.551914] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1521.179863] Lustre: lustre-MDT0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 1521.255355] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:522 to 0x2c0000401:545) [ 1521.257605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:521 to 0x280000401:545) [ 1527.241076] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004282:0x1:0x0]/32002: rc = 0 [ 1528.386134] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240003ab1:0x1:0x0]/32002 with flags 0x52: rc = 0 [ 1549.894157] Lustre: DEBUG MARKER: == sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 08:35:22 (1789302922) [ 1569.144527] Lustre: Failing over lustre-MDT0000 [ 1569.250231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1569.637154] Lustre: server umount lustre-MDT0000 complete [ 1572.988857] Lustre: Failing over lustre-MDT0001 [ 1572.990716] LustreError: 25747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789302947 with bad export cookie 10810197469120914242 [ 1573.005250] LustreError: 25747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1573.418424] Lustre: server umount lustre-MDT0001 complete [ 1578.912047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1587.445173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1594.727755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1603.614057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1614.268664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1614.297156] Lustre: lustre-MDT0000: reset Object Index mappings [ 1618.400951] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc1815f8000 x1876218946428288/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1623.047046] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1631.323920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1631.360913] Lustre: lustre-MDT0001: reset Object Index mappings [ 1631.650255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1631.667672] LustreError: Skipped 3 previous similar messages [ 1631.795639] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:554 to 0x2c0000400:577) [ 1631.817741] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:553 to 0x280000400:577) [ 1634.907222] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:586 to 0x2c0000401:609) [ 1634.914198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:585 to 0x280000401:609) [ 1636.616152] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1645.118649] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004a52:0x1:0x0]/32043: rc = 0 [ 1648.452242] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004281:0x1:0x0]/64019 with flags 0x52: rc = 0 [ 1762.838333] Lustre: DEBUG MARKER: == sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ========================================================== 08:38:56 (1789303136) [ 1792.043522] Lustre: Failing over lustre-MDT0000 [ 1792.308853] Lustre: server umount lustre-MDT0000 complete [ 1795.443944] LustreError: 19246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303169 with bad export cookie 10810197469120941472 [ 1795.448061] Lustre: Failing over lustre-MDT0001 [ 1795.465550] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1795.479213] LustreError: Skipped 1 previous similar message [ 1795.832795] Lustre: server umount lustre-MDT0001 complete [ 1801.180834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1811.257713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1820.982668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1831.110511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1842.916788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1842.957833] Lustre: lustre-MDT0000: reset Object Index mappings [ 1865.185426] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc179556680 x1876218946614784/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1865.445584] LustreError: 25256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1865.464189] LustreError: 25256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 28 previous similar messages [ 1865.554065] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1865.568864] Lustre: Skipped 1 previous similar message [ 1868.992831] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1876.639225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1876.651298] Lustre: lustre-MDT0001: reset Object Index mappings [ 1877.044679] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:618 to 0x2c0000400:641) [ 1877.077947] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:617 to 0x280000400:641) [ 1881.350367] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1882.081460] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1882.089909] Lustre: Skipped 1 previous similar message [ 1882.140778] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1882.141949] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1882.151261] Lustre: Skipped 1 previous similar message [ 1882.183067] Lustre: Skipped 14 previous similar messages [ 1882.216467] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:649 to 0x2c0000401:673) [ 1882.220572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:650 to 0x280000401:673) [ 1888.454856] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200005222:0x1:0x0]/32006: rc = 0 [ 1891.634758] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004a51:0x1:0x0]/96024 with flags 0x52: rc = 0 [ 1976.353339] Lustre: DEBUG MARKER: == sanity-scrub test 4d: FID in LMA mismatch with object FID won't block create ========================================================== 08:42:30 (1789303350) [ 1989.858881] Lustre: *** cfs_fail_loc=19b, val=0*** [ 1989.861226] Lustre: Skipped 1 previous similar message [ 1990.361933] Lustre: *** cfs_fail_loc=19b, val=0*** [ 1990.363977] Lustre: Skipped 35 previous similar messages [ 1991.372120] Lustre: *** cfs_fail_loc=19b, val=0*** [ 1991.374345] Lustre: Skipped 221 previous similar messages [ 2010.641376] Lustre: DEBUG MARKER: == sanity-scrub test 4e: FID reuse can be fixed ========== 08:43:04 (1789303384) [ 2014.983146] Lustre: *** cfs_fail_loc=1a0, val=0*** [ 2014.986174] Lustre: Skipped 197 previous similar messages [ 2015.199530] LustreError: 31725:0:(osd_compat.c:737:osd_obj_update_entry()) lustre-OST0000: the FID [0x280000401:0x321:0x0] is used by two objects: 264/3757248723 265/3727355945 [ 2026.597687] Lustre: DEBUG MARKER: == sanity-scrub test 5: OI scrub state machine =========== 08:43:20 (1789303400) [ 2043.876831] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.882183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2043.891076] Lustre: Skipped 15 previous similar messages [ 2043.900210] Lustre: Skipped 6 previous similar messages [ 2047.910688] Lustre: server umount lustre-MDT0000 complete [ 2054.114398] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2054.123633] Lustre: Skipped 1 previous similar message [ 2056.163754] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2056.171993] Lustre: Skipped 1 previous similar message [ 2057.739345] Lustre: server umount lustre-MDT0001 complete [ 2067.724334] Lustre: server umount lustre-OST0000 complete [ 2085.344220] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2085.544965] Lustre: server umount lustre-OST0001 complete [ 2090.883388] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 2097.548841] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2140.048662] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2151.197606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.510377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2151.573483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2151.733309] Lustre: lustre-MDT0000: new disk, initializing [ 2151.823738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.828842] Lustre: Skipped 7 previous similar messages [ 2151.841990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2156.675580] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2167.305107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.411255] Lustre: 75805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2167.425155] Lustre: 75805:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2167.473907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2167.480522] Lustre: Skipped 1 previous similar message [ 2167.555314] Lustre: lustre-MDT0001: new disk, initializing [ 2167.692467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2167.706877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2172.095477] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2177.099958] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2183.432160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2183.668599] Lustre: lustre-OST0000: new disk, initializing [ 2183.671739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2183.675646] Lustre: 77439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2185.447702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2185.459521] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2185.555418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2190.452722] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2201.070046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2201.253265] Lustre: lustre-OST0001: new disk, initializing [ 2201.258714] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2201.270965] Lustre: 78310:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2203.122942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2203.132249] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2203.217962] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2207.737547] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2216.614841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2219.564282] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2239.468105] Lustre: Failing over lustre-MDT0000 [ 2239.924494] Lustre: server umount lustre-MDT0000 complete [ 2244.074494] Lustre: Failing over lustre-MDT0001 [ 2244.078204] LustreError: 75798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303618 with bad export cookie 10810197469121079673 [ 2244.080897] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2244.080907] LustreError: Skipped 1 previous similar message [ 2244.111967] LustreError: 75798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 10 previous similar messages [ 2244.496707] Lustre: server umount lustre-MDT0001 complete [ 2251.281275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2261.486771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2265.574993] Lustre: 16418:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789303623/real 1789303623] req@ffff9fc177a5dc00 x1876218946926976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789303639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2265.599547] Lustre: 16418:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 26 previous similar messages [ 2269.611108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2278.469250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2289.361598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2289.376277] Lustre: lustre-MDT0000: reset Object Index mappings [ 2314.784651] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc145b82300 x1876218946930048/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2315.324386] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2320.201336] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2329.388593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2329.670154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2329.684817] LustreError: Skipped 2 previous similar messages [ 2329.824248] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:65) [ 2329.834198] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:65) [ 2334.903803] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2334.909310] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2334.929091] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2334.997665] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:65) [ 2334.999524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 2344.845949] Lustre: *** cfs_fail_loc=190, val=3*** [ 2344.848162] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000402:0x1:0x0]/32007: rc = 0 [ 2345.917035] Lustre: *** cfs_fail_loc=190, val=3*** [ 2345.923834] Lustre: Skipped 1 previous similar message [ 2346.934768] Lustre: *** cfs_fail_loc=190, val=3*** [ 2348.122152] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240000402:0x1:0x0]/32003 with flags 0x52: rc = 0 [ 2349.984619] Lustre: *** cfs_fail_loc=190, val=3*** [ 2349.986807] Lustre: Skipped 2 previous similar messages [ 2354.211501] Lustre: *** cfs_fail_loc=190, val=3*** [ 2354.219383] Lustre: Skipped 2 previous similar messages [ 2360.265969] Lustre: Failing over lustre-MDT0000 [ 2360.463832] Lustre: server umount lustre-MDT0000 complete [ 2364.080416] Lustre: Failing over lustre-MDT0001 [ 2364.321677] Lustre: server umount lustre-MDT0001 complete [ 2372.311045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2374.135798] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x96058952c2ca3a93 [ 2379.595927] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2388.176896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2388.486695] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:97) [ 2388.508943] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:97) [ 2392.828455] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2393.661164] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:97) [ 2393.661378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:97) [ 2397.995632] Lustre: Failing over lustre-MDT0000 [ 2398.341027] Lustre: server umount lustre-MDT0000 complete [ 2401.723794] Lustre: Failing over lustre-MDT0001 [ 2402.179819] Lustre: server umount lustre-MDT0001 complete [ 2410.979380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2411.085711] Lustre: *** cfs_fail_loc=190, val=3*** [ 2411.088936] Lustre: Skipped 2 previous similar messages [ 2411.945814] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x96058952c2ca40f2 [ 2411.965030] Lustre: MGC192.168.202.159@tcp: Connection restored to 0@lo (at 0@lo) [ 2411.972700] Lustre: Skipped 15 previous similar messages [ 2415.876774] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2423.388040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2423.641330] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:129) [ 2423.652321] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:129) [ 2428.198572] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2428.992713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:129) [ 2428.994723] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:129) [ 2429.152259] Lustre: *** cfs_fail_loc=190, val=3*** [ 2429.153914] Lustre: Skipped 7 previous similar messages [ 2440.004840] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240000402:0x1:0x0]/32003 with flags 0x52: rc = 0 [ 2440.015427] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000402:0x45:0x0]/142: rc = 0 [ 2451.982781] Lustre: DEBUG MARKER: == sanity-scrub test 6: OI scrub resumes from last checkpoint ========================================================== 08:50:25 (1789303825) [ 2478.404102] Lustre: Failing over lustre-MDT0000 [ 2478.912200] Lustre: server umount lustre-MDT0000 complete [ 2480.101432] LustreError: 82674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2480.120690] LustreError: 82674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 47 previous similar messages [ 2482.770622] Lustre: Failing over lustre-MDT0001 [ 2483.030925] Lustre: server umount lustre-MDT0001 complete [ 2488.304238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2496.496345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2503.904546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2512.737534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2524.022666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2524.049056] Lustre: lustre-MDT0000: reset Object Index mappings [ 2524.054295] Lustre: Skipped 1 previous similar message [ 2527.200596] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc1765ba300 x1876218947116928/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2527.218050] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [ 2533.956430] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2542.512182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2543.053451] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:193) [ 2543.076412] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:169 to 0x2c0000400:193) [ 2545.081195] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:169 to 0x2c0000401:193) [ 2545.085921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2548.546541] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2556.488214] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200001b72:0x1:0x0]/32036: rc = 0 [ 2556.488239] Lustre: *** cfs_fail_loc=190, val=2*** [ 2556.503922] Lustre: Skipped 5 previous similar messages [ 2559.836275] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240001b71:0x1:0x0]/64025 with flags 0x52: rc = 0 [ 2573.030971] Lustre: lustre-MDT0001: trigger partial OI scrub for RPC inconsistency, checking FID [0x240001b71:0x44:0x0]/257: rc = 0 [ 2573.042965] Lustre: Skipped 1 previous similar message [ 2587.072030] Lustre: Failing over lustre-MDT0000 [ 2587.105599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.312207] Lustre: server umount lustre-MDT0000 complete [ 2591.091235] Lustre: Failing over lustre-MDT0001 [ 2591.550596] Lustre: server umount lustre-MDT0001 complete [ 2602.222359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2616.824614] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x96058952c2caeb5a [ 2621.406570] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2623.457882] Lustre: *** cfs_fail_loc=190, val=3*** [ 2623.460610] Lustre: Skipped 30 previous similar messages [ 2629.167594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2629.593031] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:169 to 0x2c0000400:225) [ 2629.595906] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:225) [ 2633.501221] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2634.801553] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:169 to 0x2c0000401:225) [ 2634.820401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:225) [ 2651.928871] Lustre: DEBUG MARKER: == sanity-scrub test 7: System is available during OI scrub scanning ========================================================== 08:53:45 (1789304025) [ 2664.936304] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2685.936396] Lustre: 96488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2709.263302] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2712.989251] Lustre: 97624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2746.353692] Lustre: Failing over lustre-MDT0000 [ 2746.725366] Lustre: server umount lustre-MDT0000 complete [ 2747.361272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.382563] Lustre: Skipped 34 previous similar messages [ 2750.570436] Lustre: Failing over lustre-MDT0001 [ 2751.035808] Lustre: server umount lustre-MDT0001 complete [ 2756.943919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2768.834232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2777.180829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2787.440502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2799.661728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2799.682960] Lustre: lustre-MDT0000: reset Object Index mappings [ 2799.686538] Lustre: Skipped 1 previous similar message [ 2820.069856] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x96058952c2cbbac8 [ 2820.504402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.510749] Lustre: Skipped 13 previous similar messages [ 2825.015941] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2833.478755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2834.239949] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:266 to 0x2c0000400:289) [ 2834.240250] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:265 to 0x280000400:289) [ 2837.300091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:266 to 0x280000401:289) [ 2837.300096] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:265 to 0x2c0000401:289) [ 2838.687666] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2847.489461] Lustre: *** cfs_fail_loc=190, val=3*** [ 2847.490297] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200002b12:0x1:0x0]/32005: rc = 0 [ 2847.494646] Lustre: Skipped 9 previous similar messages [ 2850.802892] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240002b11:0x1:0x0]/32005 with flags 0x52: rc = 0 [ 2870.541303] Lustre: DEBUG MARKER: == sanity-scrub test 8: Control OI scrub manually ======== 08:57:24 (1789304244) [ 2908.179188] Lustre: Failing over lustre-MDT0000 [ 2908.669833] Lustre: server umount lustre-MDT0000 complete [ 2912.539043] Lustre: Failing over lustre-MDT0001 [ 2912.540164] LustreError: 75797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304286 with bad export cookie 10810197469121198792 [ 2912.556345] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2912.567552] LustreError: 75797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 28 previous similar messages [ 2912.589822] LustreError: Skipped 5 previous similar messages [ 2912.946554] Lustre: server umount lustre-MDT0001 complete [ 2918.306168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2928.028303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2932.705961] Lustre: 16418:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789304290/real 1789304290] req@ffff9fc1441e1180 x1876218947460992/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789304306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2932.750915] Lustre: 16418:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 62 previous similar messages [ 2935.854578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2946.409608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2959.236395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2981.925674] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc17bead500 x1876218947464704/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2981.960288] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 24 previous similar messages [ 2982.594759] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2982.597497] Lustre: Skipped 5 previous similar messages [ 2987.342251] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2995.371634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2995.591080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2995.597292] LustreError: Skipped 10 previous similar messages [ 2995.722221] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:329 to 0x280000400:353) [ 2995.724610] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:330 to 0x2c0000400:353) [ 3000.297624] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3000.810106] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3000.816755] Lustre: Skipped 5 previous similar messages [ 3000.839770] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3000.850235] Lustre: Skipped 5 previous similar messages [ 3000.917828] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:329 to 0x2c0000401:353) [ 3000.921037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:330 to 0x280000401:353) [ 3036.534301] Lustre: DEBUG MARKER: == sanity-scrub test 9: OI scrub speed control =========== 09:00:10 (1789304410) [ 3048.551107] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3065.368681] Lustre: 109131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3086.201325] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3182.634991] Lustre: Failing over lustre-MDT0000 [ 3183.244498] Lustre: server umount lustre-MDT0000 complete [ 3185.135301] LustreError: 77432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.150273] LustreError: 77432:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 29 previous similar messages [ 3186.768205] Lustre: Failing over lustre-MDT0001 [ 3187.288727] Lustre: server umount lustre-MDT0001 complete [ 3192.725774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3204.638335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3215.311212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3227.262502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3241.695627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3261.543530] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3269.670778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3269.986408] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:394 to 0x280000400:417) [ 3269.988685] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:393 to 0x2c0000400:417) [ 3274.578822] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3275.267791] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3275.273196] Lustre: Skipped 27 previous similar messages [ 3275.301607] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:393 to 0x2c0000401:417) [ 3275.302194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:394 to 0x280000401:417) [ 3335.111865] Lustre: DEBUG MARKER: == sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount (1) ========================================================== 09:05:08 (1789304708) [ 3349.764674] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3367.434551] Lustre: 117084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3367.441583] Lustre: 117084:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 3389.249970] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3524.652140] Lustre: Failing over lustre-MDT0000 [ 3525.170203] Lustre: server umount lustre-MDT0000 complete [ 3526.119884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.149681] Lustre: Skipped 14 previous similar messages [ 3529.079125] LustreError: 88007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304903 with bad export cookie 10810197469121495676 [ 3529.079371] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3529.079844] Lustre: Failing over lustre-MDT0001 [ 3529.090607] LustreError: 88007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3529.120158] LustreError: Skipped 1 previous similar message [ 3529.606289] Lustre: server umount lustre-MDT0001 complete [ 3534.976164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3544.459136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3548.449113] Lustre: 16419:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789304906/real 1789304906] req@ffff9fc175c87480 x1876218947896192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789304922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3548.485161] Lustre: 16419:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 12 previous similar messages [ 3551.829495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3561.581803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3572.938937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3572.985141] Lustre: lustre-MDT0000: reset Object Index mappings [ 3572.990480] Lustre: Skipped 5 previous similar messages [ 3574.176952] LustreError: 120809:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 3574.184511] LustreError: 120809:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9fc145b83b80 x1876218947898624/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789304948 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3574.218725] LustreError: 120809:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 3574.745427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.761612] Lustre: Skipped 5 previous similar messages [ 3579.614430] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3587.784426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3588.233208] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:458 to 0x2c0000400:481) [ 3588.233944] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:457 to 0x280000400:481) [ 3592.431399] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:458 to 0x2c0000401:481) [ 3592.444443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:457 to 0x280000401:481) [ 3593.157981] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3602.169039] Lustre: *** cfs_fail_loc=190, val=1*** [ 3602.169760] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004282:0x1:0x0]/64036: rc = 0 [ 3602.171310] Lustre: Skipped 44 previous similar messages [ 3605.503799] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004281:0x1:0x0]/32002 with flags 0x52: rc = 0 [ 3612.013484] Lustre: Failing over lustre-MDT0000 [ 3612.591914] Lustre: server umount lustre-MDT0000 complete [ 3613.154378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3613.161967] LustreError: Skipped 4 previous similar messages [ 3616.181397] Lustre: Failing over lustre-MDT0001 [ 3616.486622] Lustre: server umount lustre-MDT0001 complete [ 3625.114324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3626.272709] LustreError: 123017:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 3626.283883] LustreError: 123017:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9fc171071180 x1876218947930880/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789305000 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ptlrpcd_00_02.0' uid:0 gid:0 projid:4294967295 [ 3626.312444] LustreError: 123017:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 3626.470894] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc145b81500 x1876218947932288/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3626.772859] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3626.779481] Lustre: Skipped 2 previous similar messages [ 3631.494509] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3640.101489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3640.440960] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:457 to 0x280000400:513) [ 3640.446879] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:458 to 0x2c0000400:513) [ 3645.426625] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3645.443685] Lustre: Skipped 2 previous similar messages [ 3645.466055] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3645.472560] Lustre: Skipped 2 previous similar messages [ 3645.540404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:457 to 0x280000401:513) [ 3645.540981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:458 to 0x2c0000401:513) [ 3645.555137] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3651.565539] Lustre: Failing over lustre-MDT0000 [ 3651.842136] Lustre: server umount lustre-MDT0000 complete [ 3655.525093] Lustre: Failing over lustre-MDT0001 [ 3655.653363] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3655.658976] Lustre: Skipped 4 previous similar messages [ 3655.863972] Lustre: server umount lustre-MDT0001 complete [ 3664.224349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3665.440410] LustreError: 124980:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 3665.451681] LustreError: 124980:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9fc04c10ea00 x1876218947957120/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789305039 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ptlrpcd_00_02.0' uid:0 gid:0 projid:4294967295 [ 3665.491373] LustreError: 124980:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 3665.899317] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x96058952c2db0041 [ 3671.247591] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3679.459226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3679.931188] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:458 to 0x2c0000400:545) [ 3679.936582] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:457 to 0x280000400:545) [ 3684.335435] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3685.490787] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:458 to 0x2c0000401:545) [ 3685.490872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:457 to 0x280000401:545) [ 3698.831168] Lustre: DEBUG MARKER: == sanity-scrub test 11: OI scrub skips the new created objects only once ========================================================== 09:11:12 (1789305072) [ 3711.592784] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3730.103755] Lustre: 128090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3730.112305] Lustre: 128090:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 3748.006162] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3775.891533] Lustre: DEBUG MARKER: == sanity-scrub test 12: OI scrub can rebuild invalid /O entries ========================================================== 09:12:29 (1789305149) [ 3780.971871] Lustre: *** cfs_fail_loc=195, val=0*** [ 3785.675632] Lustre: Failing over lustre-OST0000 [ 3785.893424] Lustre: server umount lustre-OST0000 complete [ 3787.241643] LustreError: 77564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.252120] LustreError: 77564:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 37 previous similar messages [ 3795.456810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 3801.465298] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3991.247203] Lustre: DEBUG MARKER: == sanity-scrub test 13: OI scrub can rebuild missed /O entries ========================================================== 09:16:05 (1789305365) [ 3994.333502] Lustre: *** cfs_fail_loc=196, val=0*** [ 3994.337914] Lustre: Skipped 63 previous similar messages [ 3998.818273] Lustre: Failing over lustre-OST0000 [ 3998.919919] Lustre: server umount lustre-OST0000 complete [ 4006.038287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4007.863278] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4007.867631] Lustre: Skipped 22 previous similar messages [ 4010.898801] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4199.627560] Lustre: DEBUG MARKER: == sanity-scrub test 14: OI scrub can repair OST objects under lost+found ========================================================== 09:19:33 (1789305573) [ 4204.723860] Lustre: *** cfs_fail_loc=196, val=0*** [ 4204.725948] Lustre: Skipped 63 previous similar messages [ 4206.960916] Lustre: *** cfs_fail_loc=196, val=0*** [ 4206.966784] Lustre: Skipped 223 previous similar messages [ 4211.196475] Lustre: *** cfs_fail_loc=196, val=0*** [ 4211.199520] Lustre: Skipped 543 previous similar messages [ 4219.068141] Lustre: Failing over lustre-OST0000 [ 4219.167459] Lustre: server umount lustre-OST0000 complete [ 4221.411583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.423537] Lustre: Skipped 19 previous similar messages [ 4228.838547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4229.080389] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4229.092357] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4229.101858] Lustre: Skipped 3 previous similar messages [ 4234.203458] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4243.937860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4243.941119] LustreError: Skipped 3 previous similar messages [ 4243.950537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4248.200237] Lustre: server umount lustre-MDT0000 complete [ 4251.395645] LustreError: 75797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305625 with bad export cookie 10810197469122199617 [ 4251.397694] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4251.408035] LustreError: 75797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4251.430955] LustreError: Skipped 2 previous similar messages [ 4251.630610] Lustre: server umount lustre-MDT0001 complete [ 4265.053345] Lustre: server umount lustre-OST0000 complete [ 4278.582482] Lustre: server umount lustre-OST0001 complete [ 4286.935348] Lustre: DEBUG MARKER: == sanity-scrub test 15: Dryrun mode OI scrub ============ 09:21:00 (1789305660) [ 4300.261210] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4307.000809] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4341.661956] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4350.062681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.205357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4350.225181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4350.279311] Lustre: lustre-MDT0000: new disk, initializing [ 4350.332292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4350.335616] Lustre: Skipped 7 previous similar messages [ 4350.347477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4353.869572] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4361.291238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.369742] Lustre: 139369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4361.380770] Lustre: 139369:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 4361.400292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4361.404996] Lustre: Skipped 1 previous similar message [ 4361.451532] Lustre: lustre-MDT0001: new disk, initializing [ 4361.542573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4361.550393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4365.056419] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4369.060276] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4374.325168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4374.478649] Lustre: lustre-OST0000: new disk, initializing [ 4374.481816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4374.485913] Lustre: 140999:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4375.584098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4375.595837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4375.696897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4379.523974] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4388.412346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4388.495827] Lustre: lustre-OST0001: new disk, initializing [ 4388.498390] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4388.502648] Lustre: 141869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4389.870060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4389.882894] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4389.946097] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4394.160657] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4402.454463] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4405.360587] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4422.265776] Lustre: Failing over lustre-MDT0000 [ 4422.541994] Lustre: server umount lustre-MDT0000 complete [ 4423.137447] LustreError: 139380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4423.148094] LustreError: 139380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 4425.723806] Lustre: Failing over lustre-MDT0001 [ 4425.935170] Lustre: server umount lustre-MDT0001 complete [ 4430.510910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4437.305763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4444.061891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4446.183843] Lustre: 16419:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789305804/real 1789305804] req@ffff9fc04bc4ca80 x1876218948476672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789305820 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4446.206202] Lustre: 16419:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [ 4451.192609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4459.997558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4460.033114] Lustre: lustre-MDT0000: reset Object Index mappings [ 4460.040847] Lustre: Skipped 1 previous similar message [ 4474.847757] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4481.080962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4481.315681] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:65) [ 4481.332111] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:65) [ 4484.871221] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4486.644673] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4486.648594] Lustre: Skipped 4 previous similar messages [ 4486.684542] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4486.691702] Lustre: Skipped 4 previous similar messages [ 4486.731171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 4486.736473] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:65) [ 4525.140945] Lustre: DEBUG MARKER: == sanity-scrub test 16: Initial OI scrub can rebuild crashed index objects ========================================================== 09:24:59 (1789305899) [ 4536.218483] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4564.972848] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4585.584771] Lustre: Failing over lustre-MDT0000 [ 4585.602944] Lustre: *** cfs_fail_loc=199, val=0*** [ 4585.610635] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000001:0x3:0x0]: rc = 0 [ 4585.626376] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2:0x0]: rc = 0 [ 4585.635929] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x9:0x0]: rc = 0 [ 4585.643737] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xa:0x0]: rc = 0 [ 4585.649364] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xb:0x0]: rc = 0 [ 4585.659969] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xc:0x0]: rc = 0 [ 4585.666732] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xd:0x0]: rc = 0 [ 4585.672765] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xe:0x0]: rc = 0 [ 4585.678859] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x13:0x0]: rc = 0 [ 4585.685802] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x14:0x0]: rc = 0 [ 4585.692153] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x15:0x0]: rc = 0 [ 4585.696970] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x16:0x0]: rc = 0 [ 4585.703668] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x17:0x0]: rc = 0 [ 4585.712642] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x18:0x0]: rc = 0 [ 4585.719620] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x19:0x0]: rc = 0 [ 4585.725895] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1a:0x0]: rc = 0 [ 4585.732612] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1b:0x0]: rc = 0 [ 4585.738746] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1c:0x0]: rc = 0 [ 4585.745289] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1d:0x0]: rc = 0 [ 4585.751676] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1e:0x0]: rc = 0 [ 4585.758983] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1f:0x0]: rc = 0 [ 4585.765845] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x20:0x0]: rc = 0 [ 4585.775796] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x21:0x0]: rc = 0 [ 4585.781374] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x22:0x0]: rc = 0 [ 4585.786532] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x23:0x0]: rc = 0 [ 4585.792508] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x25:0x0]: rc = 0 [ 4585.798715] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x26:0x0]: rc = 0 [ 4585.804915] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x27:0x0]: rc = 0 [ 4585.810793] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x28:0x0]: rc = 0 [ 4585.815975] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x29:0x0]: rc = 0 [ 4585.829392] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2a:0x0]: rc = 0 [ 4585.835782] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2b:0x0]: rc = 0 [ 4585.840734] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2c:0x0]: rc = 0 [ 4585.846654] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2d:0x0]: rc = 0 [ 4585.854830] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2e:0x0]: rc = 0 [ 4585.860558] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2f:0x0]: rc = 0 [ 4585.865972] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x30:0x0]: rc = 0 [ 4585.871710] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x31:0x0]: rc = 0 [ 4585.876935] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x32:0x0]: rc = 0 [ 4585.884790] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x33:0x0]: rc = 0 [ 4585.896514] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x34:0x0]: rc = 0 [ 4585.901509] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x35:0x0]: rc = 0 [ 4585.908199] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x1:0x0]: rc = 0 [ 4585.913954] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x2:0x0]: rc = 0 [ 4585.919918] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x3:0x0]: rc = 0 [ 4585.925266] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20001:0x0]: rc = 0 [ 4585.930738] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20002:0x0]: rc = 0 [ 4585.940303] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20003:0x0]: rc = 0 [ 4585.946227] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x10000:0x0]: rc = 0 [ 4585.951772] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x20000:0x0]: rc = 0 [ 4585.956771] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x1010000:0x0]: rc = 0 [ 4585.962635] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x1020000:0x0]: rc = 0 [ 4585.967505] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x2010000:0x0]: rc = 0 [ 4585.972863] Lustre: 151586:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x2020000:0x0]: rc = 0 [ 4586.212593] Lustre: server umount lustre-MDT0000 complete [ 4588.962802] Lustre: Failing over lustre-MDT0001 [ 4588.972337] Lustre: *** cfs_fail_loc=199, val=0*** [ 4588.974583] Lustre: Skipped 53 previous similar messages [ 4588.976482] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000001:0x3:0x0]: rc = 0 [ 4588.983458] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x3:0x0]: rc = 0 [ 4588.996699] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x4:0x0]: rc = 0 [ 4589.007828] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x5:0x0]: rc = 0 [ 4589.020376] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x6:0x0]: rc = 0 [ 4589.029854] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x7:0x0]: rc = 0 [ 4589.037256] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x8:0x0]: rc = 0 [ 4589.047450] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xd:0x0]: rc = 0 [ 4589.055810] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xe:0x0]: rc = 0 [ 4589.061928] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xf:0x0]: rc = 0 [ 4589.068290] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x10:0x0]: rc = 0 [ 4589.081707] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x11:0x0]: rc = 0 [ 4589.098732] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x12:0x0]: rc = 0 [ 4589.111771] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x13:0x0]: rc = 0 [ 4589.122130] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x14:0x0]: rc = 0 [ 4589.129725] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x15:0x0]: rc = 0 [ 4589.135326] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x16:0x0]: rc = 0 [ 4589.140881] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x17:0x0]: rc = 0 [ 4589.146551] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x18:0x0]: rc = 0 [ 4589.150824] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x19:0x0]: rc = 0 [ 4589.159624] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1a:0x0]: rc = 0 [ 4589.173186] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1b:0x0]: rc = 0 [ 4589.187166] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1c:0x0]: rc = 0 [ 4589.194928] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1d:0x0]: rc = 0 [ 4589.205895] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1f:0x0]: rc = 0 [ 4589.213235] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x20:0x0]: rc = 0 [ 4589.223055] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x21:0x0]: rc = 0 [ 4589.232807] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x22:0x0]: rc = 0 [ 4589.239217] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x23:0x0]: rc = 0 [ 4589.245203] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x24:0x0]: rc = 0 [ 4589.250944] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x25:0x0]: rc = 0 [ 4589.256253] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x26:0x0]: rc = 0 [ 4589.261896] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x27:0x0]: rc = 0 [ 4589.267311] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x28:0x0]: rc = 0 [ 4589.273551] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x29:0x0]: rc = 0 [ 4589.279658] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2a:0x0]: rc = 0 [ 4589.285226] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2b:0x0]: rc = 0 [ 4589.291303] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2c:0x0]: rc = 0 [ 4589.296939] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2d:0x0]: rc = 0 [ 4589.304594] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2e:0x0]: rc = 0 [ 4589.310524] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2f:0x0]: rc = 0 [ 4589.316668] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x1:0x0]: rc = 0 [ 4589.322442] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x2:0x0]: rc = 0 [ 4589.328616] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x3:0x0]: rc = 0 [ 4589.334360] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20001:0x0]: rc = 0 [ 4589.339820] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20002:0x0]: rc = 0 [ 4589.345784] Lustre: 151787:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20003:0x0]: rc = 0 [ 4589.607139] Lustre: server umount lustre-MDT0001 complete [ 4596.756529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4596.842980] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace' with [0x200000003:0x13:0x0]: rc = 0 [ 4596.852042] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4596.864810] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000-MDT0000' with [0x200000005:0x20002:0x0]: rc = 0 [ 4596.877988] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0x9:0x0]: rc = 0 [ 4596.888219] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 4596.895987] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 4596.906816] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000003:0xd:0x0]: rc = 0 [ 4596.915108] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xa:0x0]: rc = 0 [ 4596.936933] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000-MDT0000' with [0x200000005:0x20003:0x0]: rc = 0 [ 4596.947524] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000003:0xc:0x0]: rc = 0 [ 4596.958796] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 4596.967838] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000003:0xe:0x0]: rc = 0 [ 4596.980183] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000-MDT0000' with [0x200000005:0x20001:0x0]: rc = 0 [ 4596.992653] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xb:0x0]: rc = 0 [ 4596.999686] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_07' with [0x200000003:0x1b:0x0]: rc = 0 [ 4597.005977] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_13' with [0x200000003:0x21:0x0]: rc = 0 [ 4597.012264] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_01' with [0x200000003:0x26:0x0]: rc = 0 [ 4597.018432] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_07' with [0x200000003:0x2c:0x0]: rc = 0 [ 4597.023488] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_06' with [0x200000003:0x2b:0x0]: rc = 0 [ 4597.031187] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_09' with [0x200000003:0x2e:0x0]: rc = 0 [ 4597.038490] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_05' with [0x200000003:0x2a:0x0]: rc = 0 [ 4597.043519] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_12' with [0x200000003:0x31:0x0]: rc = 0 [ 4597.050979] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_00' with [0x200000003:0x14:0x0]: rc = 0 [ 4597.058611] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_06' with [0x200000003:0x1a:0x0]: rc = 0 [ 4597.067969] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_02' with [0x200000003:0x27:0x0]: rc = 0 [ 4597.073198] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_14' with [0x200000003:0x33:0x0]: rc = 0 [ 4597.079340] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_03' with [0x200000003:0x17:0x0]: rc = 0 [ 4597.085725] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_15' with [0x200000003:0x23:0x0]: rc = 0 [ 4597.093699] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_15' with [0x200000003:0x34:0x0]: rc = 0 [ 4597.104571] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_10' with [0x200000003:0x2f:0x0]: rc = 0 [ 4597.124190] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_11' with [0x200000003:0x30:0x0]: rc = 0 [ 4597.131533] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_02' with [0x200000003:0x16:0x0]: rc = 0 [ 4597.145602] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_10' with [0x200000003:0x1e:0x0]: rc = 0 [ 4597.155961] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_01' with [0x200000003:0x15:0x0]: rc = 0 [ 4597.163749] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_13' with [0x200000003:0x32:0x0]: rc = 0 [ 4597.169934] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_04' with [0x200000003:0x18:0x0]: rc = 0 [ 4597.177149] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_04' with [0x200000003:0x29:0x0]: rc = 0 [ 4597.182664] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_08' with [0x200000003:0x1c:0x0]: rc = 0 [ 4597.187964] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_09' with [0x200000003:0x1d:0x0]: rc = 0 [ 4597.192374] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_14' with [0x200000003:0x22:0x0]: rc = 0 [ 4597.196924] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_03' with [0x200000003:0x28:0x0]: rc = 0 [ 4597.202477] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_11' with [0x200000003:0x1f:0x0]: rc = 0 [ 4597.208288] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_00' with [0x200000003:0x25:0x0]: rc = 0 [ 4597.215446] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_05' with [0x200000003:0x19:0x0]: rc = 0 [ 4597.225348] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_08' with [0x200000003:0x2d:0x0]: rc = 0 [ 4597.233396] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_12' with [0x200000003:0x20:0x0]: rc = 0 [ 4597.241149] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x2:0x0]: rc = 0 [ 4597.248256] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 4597.257645] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 4597.263638] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 4597.268710] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 4597.272802] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 4597.276685] Lustre: 152284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 4602.224736] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4607.549277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4607.594170] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace' with [0x200000003:0xd:0x0]: rc = 0 [ 4607.599875] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4607.607144] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 4607.613549] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1020000-MDT0001' with [0x200000005:0x20002:0x0]: rc = 0 [ 4607.619948] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x3:0x0]: rc = 0 [ 4607.625391] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1020000' with [0x200000003:0x7:0x0]: rc = 0 [ 4607.632580] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 4607.642933] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x20000-MDT0001' with [0x200000005:0x20001:0x0]: rc = 0 [ 4607.655639] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 4607.665236] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2020000' with [0x200000003:0x8:0x0]: rc = 0 [ 4607.675629] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x4:0x0]: rc = 0 [ 4607.685247] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2020000-MDT0001' with [0x200000005:0x20003:0x0]: rc = 0 [ 4607.689866] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x20000' with [0x200000003:0x6:0x0]: rc = 0 [ 4607.701337] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x5:0x0]: rc = 0 [ 4607.709872] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_07' with [0x200000003:0x26:0x0]: rc = 0 [ 4607.716791] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_09' with [0x200000003:0x17:0x0]: rc = 0 [ 4607.724667] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_03' with [0x200000003:0x11:0x0]: rc = 0 [ 4607.733617] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_04' with [0x200000003:0x12:0x0]: rc = 0 [ 4607.740455] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_15' with [0x200000003:0x1d:0x0]: rc = 0 [ 4607.746588] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_03' with [0x200000003:0x22:0x0]: rc = 0 [ 4607.755553] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_04' with [0x200000003:0x23:0x0]: rc = 0 [ 4607.769195] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_00' with [0x200000003:0x1f:0x0]: rc = 0 [ 4607.776754] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_02' with [0x200000003:0x10:0x0]: rc = 0 [ 4607.784932] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_13' with [0x200000003:0x2c:0x0]: rc = 0 [ 4607.792630] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_09' with [0x200000003:0x28:0x0]: rc = 0 [ 4607.800592] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_05' with [0x200000003:0x24:0x0]: rc = 0 [ 4607.809371] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_05' with [0x200000003:0x13:0x0]: rc = 0 [ 4607.816676] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_10' with [0x200000003:0x18:0x0]: rc = 0 [ 4607.828873] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_13' with [0x200000003:0x1b:0x0]: rc = 0 [ 4607.842048] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_02' with [0x200000003:0x21:0x0]: rc = 0 [ 4607.849846] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_06' with [0x200000003:0x14:0x0]: rc = 0 [ 4607.864084] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_01' with [0x200000003:0xf:0x0]: rc = 0 [ 4607.876133] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_12' with [0x200000003:0x2b:0x0]: rc = 0 [ 4607.885956] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_12' with [0x200000003:0x1a:0x0]: rc = 0 [ 4607.893339] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_15' with [0x200000003:0x2e:0x0]: rc = 0 [ 4607.898991] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_11' with [0x200000003:0x2a:0x0]: rc = 0 [ 4607.906262] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_07' with [0x200000003:0x15:0x0]: rc = 0 [ 4607.912268] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_14' with [0x200000003:0x1c:0x0]: rc = 0 [ 4607.918826] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_06' with [0x200000003:0x25:0x0]: rc = 0 [ 4607.928078] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_14' with [0x200000003:0x2d:0x0]: rc = 0 [ 4607.936435] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_08' with [0x200000003:0x27:0x0]: rc = 0 [ 4607.947108] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_11' with [0x200000003:0x19:0x0]: rc = 0 [ 4607.953390] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_01' with [0x200000003:0x20:0x0]: rc = 0 [ 4607.958302] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_10' with [0x200000003:0x29:0x0]: rc = 0 [ 4607.963233] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_00' with [0x200000003:0xe:0x0]: rc = 0 [ 4607.973437] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4607.977321] Lustre: Skipped 8 previous similar messages [ 4607.980628] Lustre: 153028:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_08' with [0x200000003:0x16:0x0]: rc = 0 [ 4608.180209] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:105 to 0x2c0000400:129) [ 4608.188270] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:106 to 0x280000400:129) [ 4611.016948] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4612.505632] Lustre: lustre-MDT0000: Denying connection for new client 95c0ff83-754a-46ea-86e4-21d1f20a9314 (at 192.168.202.59@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4613.663225] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 4613.666839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:129) [ 4623.391556] Lustre: DEBUG MARKER: == sanity-scrub test 17a: ENOSPC on OI insert shouldn't leak inodes ========================================================== 09:26:37 (1789305997) [ 4624.135546] Lustre: *** cfs_fail_loc=19d, val=0*** [ 4624.137085] Lustre: Skipped 123 previous similar messages [ 4625.649869] Lustre: Failing over lustre-MDT0000 [ 4625.898261] Lustre: server umount lustre-MDT0000 complete [ 4636.716762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.148824] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4642.368698] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:161) [ 4642.368749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:161) [ 4642.601535] Lustre: DEBUG MARKER: == sanity-scrub test 17b: ENOSPC on .. insertion shouldn't leak inodes ========================================================== 09:26:56 (1789306016) [ 4643.271288] Lustre: *** cfs_fail_loc=19e, val=0*** [ 4644.474603] Lustre: Failing over lustre-MDT0000 [ 4644.635756] Lustre: server umount lustre-MDT0000 complete [ 4654.209072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.471682] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4659.753448] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:193) [ 4659.754807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:193) [ 4660.018538] Lustre: DEBUG MARKER: == sanity-scrub test 18: test mount -o resetoi to recreate OI files ========================================================== 09:27:13 (1789306033) [ 4676.621107] Lustre: Failing over lustre-MDT0000 [ 4676.852443] Lustre: server umount lustre-MDT0000 complete [ 4679.467505] Lustre: Failing over lustre-MDT0001 [ 4679.665640] Lustre: server umount lustre-MDT0001 complete [ 4686.057666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4693.579457] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4699.953551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4700.168723] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:193) [ 4700.170688] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4703.016282] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4704.649052] Lustre: lustre-MDT0000: Denying connection for new client aa39f793-5b3d-46ad-926a-818dd3d6776f (at 192.168.202.59@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4705.309748] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:257) [ 4705.313877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:234 to 0x280000401:257) [ 4712.845995] Lustre: Failing over lustre-MDT0000 [ 4713.022277] Lustre: server umount lustre-MDT0000 complete [ 4715.470095] Lustre: Failing over lustre-MDT0001 [ 4715.491832] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4715.498267] Lustre: Skipped 5 previous similar messages [ 4715.673609] Lustre: server umount lustre-MDT0001 complete [ 4721.610592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4725.730610] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x96058952c2ddfecf [ 4729.083465] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4734.850104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4735.158249] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:225) [ 4735.158337] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:225) [ 4736.206966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:289) [ 4736.208770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:234 to 0x280000401:289) [ 4738.155939] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4744.673436] Lustre: DEBUG MARKER: == sanity-scrub test 19: LFSCK can fix multiple linked files on OST ========================================================== 09:28:38 (1789306118) [ 4750.824717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4755.340518] Lustre: server umount lustre-MDT0000 complete [ 4758.059286] Lustre: server umount lustre-MDT0001 complete [ 4770.363982] Lustre: server umount lustre-OST0000 complete [ 4783.323528] Lustre: server umount lustre-OST0001 complete [ 4787.846449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4794.090987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4809.568420] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4814.688295] LustreError: 161793:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 4844.948745] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4848.338770] Lustre: Failing over lustre-OST0000 [ 4848.406411] Lustre: server umount lustre-OST0000 complete [ 4852.376698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4858.992719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4874.530356] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4879.650067] LustreError: 163316:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 4909.574150] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4914.429536] Lustre: DEBUG MARKER: == sanity-scrub test 20: Don't trigger OI scrub for irreparable oi repeatedly ========================================================== 09:31:28 (1789306288) [ 4923.534994] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4930.891310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.224092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:385) [ 4934.270526] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4940.398894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4940.737561] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:257) [ 4944.532612] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4946.764695] Lustre: 166220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4946.769699] Lustre: 166220:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 4957.404802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4957.517579] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4957.522358] Lustre: Skipped 17 previous similar messages [ 4961.874188] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4962.796930] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:321) [ 4962.800720] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:257) [ 4967.169957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4971.380357] Lustre: *** cfs_fail_loc=193, val=0*** [ 4972.618927] Lustre: Failing over lustre-MDT0000 [ 4972.897488] Lustre: server umount lustre-MDT0000 complete [ 4973.032322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4973.043169] Lustre: Skipped 39 previous similar messages [ 4978.452811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4978.529757] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4978.532516] LustreError: Skipped 7 previous similar messages [ 4978.623220] Lustre: *** cfs_fail_loc=193, val=0*** [ 4978.624442] Lustre: Skipped 1 previous similar message [ 4978.695456] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4978.706316] Lustre: Skipped 6 previous similar messages [ 4981.512297] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4983.790909] Lustre: *** cfs_fail_loc=193, val=0*** [ 4983.796397] Lustre: Skipped 4 previous similar messages [ 4983.864639] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:353) [ 4983.864649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:417) [ 4984.530987] Lustre: *** cfs_fail_loc=19f, val=0*** [ 4984.533153] Lustre: Skipped 58 previous similar messages [ 4984.533822] Lustre: lustre-MDT0000: trigger OI scrub by RPC for [0x200003ab1:0x2:0x0]/16843009 with flags 0x4a: rc = 0 [ 4996.570539] Lustre: DEBUG MARKER: == sanity-scrub test 21: don't hang MDS recovery when failed to get update log ========================================================== 09:32:50 (1789306370) [ 4998.538386] Lustre: Failing over lustre-MDT0000 [ 4998.718221] Lustre: server umount lustre-MDT0000 complete [ 5001.301912] LustreError: 163322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306375 with bad export cookie 10810197469122418941 [ 5001.307206] LustreError: 163322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 14 previous similar messages [ 5001.307221] Lustre: Failing over lustre-MDT0001 [ 5001.491546] Lustre: server umount lustre-MDT0001 complete [ 5003.211289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5007.790150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.427401] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc04c1d9f80 x1876218948953856/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5011.445383] LustreError: 16415:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 5014.730917] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5020.269408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.407217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 5020.414077] LustreError: Skipped 7 previous similar messages [ 5023.686862] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5026.097697] LustreError: 170716:0:(update_trans.c:1081:top_trans_stop()) lustre-MDT0000-osp-MDT0001: stop trans failed: rc = -20 [ 5026.123579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:449) [ 5026.123601] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:385) [ 5026.163354] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:289) [ 5026.165860] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:289) [ 5028.122290] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 200 [ 5032.620232] Lustre: DEBUG MARKER: == sanity-scrub test 22: LFSCK can recreate or fix the LASTID on MDT/OST ========================================================== 09:33:26 (1789306406) [ 5036.008680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5036.013481] Lustre: Skipped 7 previous similar messages [ 5041.267972] Lustre: server umount lustre-MDT0000 complete [ 5043.949717] Lustre: server umount lustre-MDT0001 complete [ 5056.979733] Lustre: server umount lustre-OST0000 complete [ 5069.835283] Lustre: server umount lustre-OST0001 complete [ 5073.945470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5081.244808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5081.541159] LustreError: 173075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5081.553427] LustreError: 173075:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 93 previous similar messages [ 5084.485502] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5087.711944] Lustre: Failing over lustre-MDT0000 [ 5087.717370] LustreError: 173108:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5087.722511] LustreError: 173108:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5087.732900] LustreError: 173108:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 5087.880492] Lustre: server umount lustre-MDT0000 complete [ 5091.475494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5100.062192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5103.387432] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5108.728116] Lustre: DEBUG MARKER: === sanity-scrub: start setup 09:34:42 (1789306482) === [ 5110.272608] LustreError: 174744:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5110.281819] LustreError: 174744:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5110.286825] LustreError: 174744:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 5110.589072] Lustre: server umount lustre-MDT0000 complete [ 5132.691355] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5137.990874] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5170.577072] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5179.467401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5179.695292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5179.716915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5179.765776] Lustre: lustre-MDT0000: new disk, initializing [ 5179.825791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5182.741758] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5191.734155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5191.825281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5191.827377] Lustre: Skipped 1 previous similar message [ 5191.865900] Lustre: lustre-MDT0001: new disk, initializing [ 5191.920176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5191.930830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5195.141353] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5198.984496] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5206.356539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5206.502743] Lustre: lustre-OST0000: new disk, initializing [ 5206.505563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5206.509977] Lustre: 182034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5208.014382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5208.025111] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5208.086827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5210.866390] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5219.621392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5219.692523] Lustre: lustre-OST0001: new disk, initializing [ 5219.694410] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5219.697538] Lustre: 183060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5220.924840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5220.933994] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5220.997572] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5223.545646] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5230.152364] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5232.443405] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5236.028857] Lustre: DEBUG MARKER: === sanity-scrub: finish setup 09:36:50 (1789306610) === [ 5236.952830] Lustre: DEBUG MARKER: == sanity-scrub test complete, duration 4941 sec ========= 09:36:51 (1789306611) [ 5237.871818] Lustre: DEBUG MARKER: === sanity-scrub: start cleanup 09:36:52 (1789306612) === [ 5239.610691] Lustre: DEBUG MARKER: === sanity-scrub: finish cleanup 09:36:53 (1789306613) === [ 5241.313143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5241.315374] Lustre: Skipped 7 previous similar messages [ 5247.437636] Lustre: server umount lustre-MDT0000 complete [ 5252.938252] Lustre: server umount lustre-MDT0001 complete [ 5268.050238] Lustre: server umount lustre-OST0000 complete [ 5272.033106] Lustre: 16418:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789306630/real 1789306630] req@ffff9fc0493af480 x1876218949058432/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789306646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5272.055345] Lustre: 16418:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 35 previous similar messages [ 5273.122782] Lustre: server umount lustre-OST0001 complete [ 5280.408618] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5281.684417] Key type lgssc unregistered [ 5281.828432] LNet: 186467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5281.832480] LNetError: 186467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5281.839583] LNet: Removed LNI 192.168.202.159@tcp [ 5282.251148] Key type .llcrypt unregistered [ 5282.253296] Key type ._llcrypt unregistered