[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 559111464 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002433] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007818] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012056] Yama: becoming mindful. [ 0.013032] SELinux: Initializing. [ 0.014088] *** VALIDATE selinux *** [ 0.022748] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026806] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028036] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030141] *** VALIDATE tmpfs *** [ 0.031464] *** VALIDATE proc *** [ 0.033037] *** VALIDATE cgroup *** [ 0.033890] *** VALIDATE cgroup2 *** [ 0.035005] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038025] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042535] debug: unmapping init [mem 0xffffffff9be59000-0xffffffff9be60fff] [ 0.044165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045611] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.047985] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050008] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052249] rcu: Hierarchical SRCU implementation. [ 0.054296] smp: Bringing up secondary CPUs ... [ 0.055510] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.059239] smp: Brought up 1 node, 4 CPUs [ 0.060996] smpboot: Max logical packages: 1 [ 0.062013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135000] node 0 deferred pages initialised in 70ms [ 0.136141] devtmpfs: initialized [ 0.136794] x86/mm: Memory block size: 128MB [ 0.138141] gcov: version magic: 0x41383552 [ 0.140222] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141055] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142205] pinctrl core: initialized pinctrl subsystem [ 0.143167] [ 0.143680] ************************************************************* [ 0.144013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145011] ** ** [ 0.146011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147009] ** ** [ 0.148009] ** This means that this kernel is built to expose internal ** [ 0.149008] ** IOMMU data structures, which may compromise security on ** [ 0.150007] ** your system. ** [ 0.151011] ** ** [ 0.152008] ** If you see this message and you are not debugging the ** [ 0.153009] ** kernel, report this immediately to your vendor! ** [ 0.154007] ** ** [ 0.155008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156007] ************************************************************* [ 0.157499] NET: Registered protocol family 16 [ 0.158355] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.159056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.160063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.161481] cpuidle: using governor menu [ 0.162000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.164412] PCI: Using configuration type 1 for base access [ 0.166124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.175098] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.176017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.178170] cryptd: max_cpu_qlen set to 1000 [ 0.180215] ACPI: Added _OSI(Module Device) [ 0.181013] ACPI: Added _OSI(Processor Device) [ 0.182014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.183008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.186604] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.188525] ACPI: Interpreter enabled [ 0.190061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.192005] ACPI: Using IOAPIC for interrupt routing [ 0.193117] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.196355] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.207230] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.209039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.212019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.214077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.217964] acpiphp: Slot [2] registered [ 0.219098] acpiphp: Slot [5] registered [ 0.221120] acpiphp: Slot [6] registered [ 0.222074] acpiphp: Slot [7] registered [ 0.223102] acpiphp: Slot [8] registered [ 0.225140] acpiphp: Slot [9] registered [ 0.226092] acpiphp: Slot [10] registered [ 0.228110] acpiphp: Slot [3] registered [ 0.229090] acpiphp: Slot [4] registered [ 0.230124] acpiphp: Slot [11] registered [ 0.232100] acpiphp: Slot [12] registered [ 0.233077] acpiphp: Slot [13] registered [ 0.235091] acpiphp: Slot [14] registered [ 0.236111] acpiphp: Slot [15] registered [ 0.237114] acpiphp: Slot [16] registered [ 0.239115] acpiphp: Slot [17] registered [ 0.241087] acpiphp: Slot [18] registered [ 0.242083] acpiphp: Slot [19] registered [ 0.243094] acpiphp: Slot [20] registered [ 0.245097] acpiphp: Slot [21] registered [ 0.246089] acpiphp: Slot [22] registered [ 0.248125] acpiphp: Slot [23] registered [ 0.249094] acpiphp: Slot [24] registered [ 0.251108] acpiphp: Slot [25] registered [ 0.252087] acpiphp: Slot [26] registered [ 0.253141] acpiphp: Slot [27] registered [ 0.255175] acpiphp: Slot [28] registered [ 0.256080] acpiphp: Slot [29] registered [ 0.257080] acpiphp: Slot [30] registered [ 0.258075] acpiphp: Slot [31] registered [ 0.260126] PCI host bridge to bus 0000:00 [ 0.261013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.263016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.265014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.267024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.270022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.273019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.274128] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.277027] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.279971] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.289009] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.294026] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.295011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.297010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.298011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.300392] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.301587] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.304035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.306579] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.310011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.319702] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.323012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.328381] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.336015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.342014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.355014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.364829] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.369014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.374014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.389016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.397836] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.402012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.405012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.415013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.422189] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.425015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.432016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.444020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.454444] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.461011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.465014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.477015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.488088] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.494071] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.500013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.519017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.529321] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.533335] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.535332] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.537369] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.539275] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.543125] iommu: Default domain type: Passthrough [ 0.547484] SCSI subsystem initialized [ 0.550167] ACPI: bus type USB registered [ 0.551087] usbcore: registered new interface driver usbfs [ 0.553108] usbcore: registered new interface driver hub [ 0.555121] usbcore: registered new device driver usb [ 0.557188] pps_core: LinuxPPS API ver. 1 registered [ 0.559012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.562083] PTP clock support registered [ 0.564233] EDAC MC: Ver: 3.0.0 [ 0.565580] PCI: Using ACPI for IRQ routing [ 0.566371] NetLabel: Initializing [ 0.567000] NetLabel: domain hash size = 128 [ 0.569025] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.573085] NetLabel: unlabeled traffic allowed by default [ 0.575115] vgaarb: loaded [ 0.577264] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.579011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.586060] clocksource: Switched to clocksource kvm-clock [ 0.692097] VFS: Disk quotas dquot_6.6.0 [ 0.693741] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.696655] *** VALIDATE ramfs *** [ 0.698457] *** VALIDATE hugetlbfs *** [ 0.700068] pnp: PnP ACPI init [ 0.703654] pnp: PnP ACPI: found 6 devices [ 0.720800] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.724297] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.726638] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.729195] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.731412] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.733752] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.736479] NET: Registered protocol family 2 [ 0.738730] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.743808] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.747344] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.752612] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.755695] TCP: Hash tables configured (established 65536 bind 65536) [ 0.758566] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.761384] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.763926] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.766833] NET: Registered protocol family 1 [ 0.769515] RPC: Registered named UNIX socket transport module. [ 0.771509] RPC: Registered udp transport module. [ 0.772625] RPC: Registered tcp transport module. [ 0.773826] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.775218] NET: Registered protocol family 44 [ 0.777173] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.779263] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.781814] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.784680] PCI: CLS 0 bytes, default 64 [ 0.786570] Unpacking initramfs... [ 2.252478] debug: unmapping init [mem 0xffff990fbcc54000-0xffff990fbffbffff] [ 2.256227] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.258069] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.260645] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.725335] Initialise system trusted keyrings [ 2.727068] Key type blacklist registered [ 2.729224] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.739180] zbud: loaded [ 2.742214] *** VALIDATE nfs *** [ 2.743113] *** VALIDATE nfs4 *** [ 2.744307] pstore: using deflate compression [ 2.756214] Platform Keyring initialized [ 2.848126] NET: Registered protocol family 38 [ 2.849424] Key type asymmetric registered [ 2.850894] Asymmetric key parser 'x509' registered [ 2.852904] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.855347] io scheduler mq-deadline registered [ 2.856527] io scheduler kyber registered [ 2.857722] io scheduler bfq registered [ 2.859432] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.862425] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.865386] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.867666] ACPI: Power Button [PWRF] [ 2.871934] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.877510] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.887450] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.893995] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.905897] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.931769] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.959064] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.964439] Non-volatile memory driver v1.3 [ 2.965962] Linux agpgart interface v0.103 [ 2.996171] virtio_blk virtio1: [vda] 146640 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.998077] vda: detected capacity change from 0 to 75079680 [ 3.013280] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.016030] vdb: detected capacity change from 0 to 1073741824 [ 3.035359] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.038063] vdc: detected capacity change from 0 to 2621440000 [ 3.054354] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.057190] vdd: detected capacity change from 0 to 2621440000 [ 3.078466] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.081403] vde: detected capacity change from 0 to 4294967296 [ 3.099652] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.101673] vdf: detected capacity change from 0 to 4294967296 [ 3.109612] libphy: Fixed MDIO Bus: probed [ 3.121145] usbcore: registered new interface driver usbserial_generic [ 3.123077] usbserial: USB Serial support registered for generic [ 3.125195] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.128626] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.130218] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.132873] mousedev: PS/2 mouse device common for all mice [ 3.135659] rtc_cmos 00:05: RTC can wake from S4 [ 3.140206] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.140405] rtc_cmos 00:05: registered as rtc0 [ 3.144128] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.147341] intel_pstate: CPU model not supported [ 3.149829] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.152993] hid: raw HID events driver (C) Jiri Kosina [ 3.155201] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.155269] usbcore: registered new interface driver usbhid [ 3.159457] usbhid: USB HID core driver [ 3.161052] drop_monitor: Initializing network drop monitor service [ 3.163614] Initializing XFRM netlink socket [ 3.165889] NET: Registered protocol family 10 [ 3.169393] Segment Routing with IPv6 [ 3.171262] NET: Registered protocol family 17 [ 3.173466] mpls_gso: MPLS GSO support [ 3.178390] RAS: Correctable Errors collector initialized. [ 3.179749] AVX version of gcm_enc/dec engaged. [ 3.180872] AES CTR mode by8 optimization enabled [ 3.238132] sched_clock: Marking stable (3238108181, 0)->(4162746071, -924637890) [ 3.241430] registered taskstats version 1 [ 3.243704] Loading compiled-in X.509 certificates [ 3.245424] zswap: loaded using pool lzo/zbud [ 3.264203] Key type big_key registered [ 3.274941] Key type encrypted registered [ 3.276377] ima: No TPM chip found, activating TPM-bypass! [ 3.278469] ima: Allocated hash algorithm: sha1 [ 3.279963] ima: No architecture policies found [ 3.281522] evm: Initialising EVM extended attributes: [ 3.283439] evm: security.selinux [ 3.284581] evm: security.ima [ 3.285684] evm: security.capability [ 3.287056] evm: HMAC attrs: 0x1 [ 3.289226] rtc_cmos 00:05: setting system clock to 2026-08-31 21:21:00 UTC (1788211260) [ 3.295090] debug: unmapping init [mem 0xffffffff9ce03000-0xffffffff9cffffff] [ 3.298096] debug: unmapping init [mem 0xffffffff9bb82000-0xffffffff9be58fff] [ 3.307088] Write protecting the kernel read-only data: 28672k [ 3.310248] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 3.312593] debug: unmapping init [mem 0xffffffff9ab14000-0xffffffff9abfffff] [ 3.345563] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.353704] systemd[1]: Detected virtualization kvm. [ 3.355027] systemd[1]: Detected architecture x86-64. [ 3.356886] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.389550] systemd[1]: No hostname configured. [ 3.391373] systemd[1]: Set hostname to . [ 3.393208] random: systemd: uninitialized urandom read (16 bytes read) [ 3.395573] systemd[1]: Initializing machine ID from random generator. [ 3.428459] random: ln: uninitialized urandom read (6 bytes read) [ 3.498747] random: systemd: uninitialized urandom read (16 bytes read) [ 3.500398] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.504935] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.508093] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.011251] device-mapper: uevent: version 1.0.3 [ 4.012710] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.680192] virtio_net virtio0 ens2: renamed from eth0 [ 4.711912] random: fast init done [ 4.718338] scsi host0: ata_piix [ 4.740954] scsi host1: ata_piix [ 4.742675] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.745076] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.121420] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.792127] random: crng init done [ 9.793668] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.130233] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.394505] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.674044] SELinux: Disabled at runtime. [ 11.734886] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.741279] systemd[1]: Detected virtualization kvm. [ 11.742632] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.191984] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.195351] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.200158] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.206317] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.209596] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.218484] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.226973] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ 12.316964] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.732271] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.061040] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.226645] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.309819] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.330719] EDAC sbridge: Ver: 1.1.2 [ 14.969406] Key type dns_resolver registered [ 15.274729] NFS: Registering the id_resolver key type [ 15.276730] Key type id_resolver registered [ 15.278354] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg219-server login: [ 33.674290] spl: loading out-of-tree module taints kernel. [ 40.099566] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 56.579484] Key type ._llcrypt registered [ 56.581963] Key type .llcrypt registered [ 56.715935] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 73.395479] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 75.147738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.165253] alg: No test for adler32 (adler32-zlib) [ 76.700466] Lustre: Lustre: Build Version: 2.17.57_85_gb71fc30 [ 77.823378] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 79.639220] Key type lgssc registered [ 81.007308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 84.193067] hrtimer: interrupt took 3343740 ns [ 90.516710] vdc: vdc1 vdc9 [ 100.243045] vdd: vdd1 vdd9 [ 110.966801] vde: vde1 vde9 [ 122.946600] vdf: vdf1 vdf9 [ 143.739411] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 154.871908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.414930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 156.895079] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 156.996745] Lustre: lustre-MDT0000: new disk, initializing [ 157.400506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.492954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 162.370660] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.299728] Lustre: 6429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 172.364963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 172.368165] Lustre: Skipped 1 previous similar message [ 172.422955] Lustre: lustre-MDT0001: new disk, initializing [ 172.702979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 172.766822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 172.774023] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.366092] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.301596] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.441974] Lustre: lustre-OST0000: new disk, initializing [ 189.452431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 189.602225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.565662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.573803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.903893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 195.986956] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.890745] Lustre: lustre-OST0001: new disk, initializing [ 206.894977] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 207.024942] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.789343] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.147323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 216.160474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 216.525701] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 226.415548] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 237.904371] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 245.081177] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing check_logdir /tmp/testlogs/ [ 250.551429] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing yml_node [ 255.647931] Lustre: DEBUG MARKER: Client: 2.17.57.85 [ 258.425861] Lustre: DEBUG MARKER: MDS: 2.17.57.85 [ 261.626837] Lustre: DEBUG MARKER: OSS: 2.17.57.85 [ 263.650926] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-scrub ============----- Mon Aug 31 17:25:19 EDT 2026 [ 284.780730] Lustre: DEBUG MARKER: excepting tests: [ 295.528213] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 303.075931] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.077317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.087214] Lustre: Skipped 1 previous similar message [ 303.098362] Lustre: Skipped 3 previous similar messages [ 304.344477] Lustre: server umount lustre-MDT0000 complete [ 308.700553] LustreError: 8097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788211565 with bad export cookie 15560189118844547865 [ 308.701812] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 308.709784] LustreError: 8097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 309.357422] Lustre: server umount lustre-MDT0001 complete [ 324.515218] Lustre: server umount lustre-OST0000 complete [ 339.621409] Lustre: server umount lustre-OST0001 complete [ 352.517550] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 355.219833] Key type lgssc unregistered [ 355.627306] LNet: 13309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.632053] LNetError: 13309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 355.658648] LNet: Removed LNI 192.168.202.119@tcp [ 356.466913] Key type .llcrypt unregistered [ 356.469651] Key type ._llcrypt unregistered [ 374.044945] Key type ._llcrypt registered [ 374.047755] Key type .llcrypt registered [ 374.096520] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 389.029621] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 390.104387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 390.318775] alg: No test for adler32 (adler32-zlib) [ 391.412273] Lustre: Lustre: Build Version: 2.17.57_85_gb71fc30 [ 391.588227] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 393.223171] Key type lgssc registered [ 394.035435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 401.637723] vdc: vdc1 vdc9 [ 412.628561] vdd: vdd1 vdd9 [ 423.290718] vde: vde1 vde9 [ 434.510736] vdf: vdf1 vdf9 [ 454.251226] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 463.361481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 464.817961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 464.880307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 464.935216] Lustre: lustre-MDT0000: new disk, initializing [ 465.210864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.257315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 471.573378] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 486.168263] Lustre: 17728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 486.286956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 486.291223] Lustre: Skipped 1 previous similar message [ 486.362895] Lustre: lustre-MDT0001: new disk, initializing [ 486.628595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 486.690769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 486.721052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 492.380492] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 498.085191] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 506.533576] Lustre: lustre-OST0000: new disk, initializing [ 506.536822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 506.628870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.107938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 507.129802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 507.642251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 514.020254] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 526.299337] Lustre: lustre-OST0001: new disk, initializing [ 526.302886] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 526.391590] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 532.124104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 532.133782] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 532.509589] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 534.333945] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 546.389645] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 554.158447] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 562.658177] Lustre: DEBUG MARKER: === sanity-scrub: finish setup 17:30:17 (1788211817) === [ 565.092173] Lustre: DEBUG MARKER: == sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 17:30:20 (1788211820) [ 594.387581] Lustre: Failing over lustre-MDT0000 [ 594.415933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -19 [ 594.436168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.479563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 594.832781] Lustre: server umount lustre-MDT0000 complete [ 599.011203] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.217622] LustreError: 17720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788211857 with bad export cookie 7899596791058457028 [ 600.219656] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 600.242139] LustreError: 17720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 600.274987] Lustre: Failing over lustre-MDT0001 [ 600.968758] Lustre: server umount lustre-MDT0001 complete [ 620.512935] Lustre: 14731:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788211861/real 1788211861] req@ffff9910103e0700 x1875075817151104/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788211877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 620.542815] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.556470] Lustre: Skipped 2 previous similar messages [ 621.535159] Lustre: 14734:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788211862/real 1788211862] req@ffff99103c81ad80 x1875075817151232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788211878 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 621.574983] Lustre: 14734:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 625.696150] Lustre: 14731:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788211866/real 1788211866] req@ffff9910103e0000 x1875075817151616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788211882 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 625.704459] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6da1016d77db2188 [ 625.739207] Lustre: 14731:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 625.753377] Lustre: MGC192.168.202.119@tcp: Connection restored to 0@lo (at 0@lo) [ 625.978450] LustreError: 19380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.993777] LustreError: 19380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 626.239232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.414691] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 630.187931] LustreError: 19381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.219207] LustreError: 19381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 630.879120] Lustre: 14732:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788211872/real 1788211872] req@ffff99100713a680 x1875075817152128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788211888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 630.916728] Lustre: 14732:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 632.412339] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 635.302517] Lustre: 14733:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788211876/real 1788211876] req@ffff99103d23ce00 x1875075817152768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788211892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 635.356890] Lustre: 14733:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 635.381705] LustreError: 22310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.412295] LustreError: 22310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 639.522831] LustreError: 19380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.536143] LustreError: 19380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 639.543477] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 641.147568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 641.302985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 641.398911] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:65) [ 641.402642] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:65) [ 646.151535] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 646.652818] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 646.665311] Lustre: Skipped 1 previous similar message [ 646.726439] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 646.783072] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 646.811933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 646.815360] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:65) [ 661.578441] Lustre: DEBUG MARKER: == sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 17:31:57 (1788211917) [ 686.687829] Lustre: Failing over lustre-MDT0000 [ 687.097704] Lustre: server umount lustre-MDT0000 complete [ 687.591694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.595344] LustreError: 23367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 687.619370] Lustre: Skipped 3 previous similar messages [ 687.639391] LustreError: 23367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 692.113105] Lustre: Failing over lustre-MDT0001 [ 692.114526] LustreError: 21112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788211949 with bad export cookie 7899596791058473352 [ 692.122698] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 692.141028] LustreError: 21112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 692.555150] Lustre: server umount lustre-MDT0001 complete [ 708.063258] Lustre: 14733:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788211949/real 1788211949] req@ffff9910382ab800 x1875075817267456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788211965 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 708.100200] Lustre: 14733:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 708.106048] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.125964] Lustre: Skipped 1 previous similar message [ 717.727649] LustreError: 19381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 717.886995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.940381] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 723.252168] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 727.013450] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 727.027990] Lustre: Skipped 2 previous similar messages [ 733.901326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 734.251037] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 734.576627] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:106 to 0x280000400:129) [ 734.603553] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:105 to 0x2c0000400:129) [ 739.298608] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 739.312073] Lustre: Skipped 1 previous similar message [ 739.459586] Lustre: lustre-MDT0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 739.489823] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:106 to 0x2c0000401:129) [ 739.510694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 740.026672] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 745.614457] Lustre: *** cfs_fail_loc=193, val=0*** [ 750.830660] Lustre: Failing over lustre-MDT0000 [ 751.139195] Lustre: server umount lustre-MDT0000 complete [ 754.662109] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.663873] LustreError: 19380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 754.689063] Lustre: Skipped 4 previous similar messages [ 754.734607] LustreError: 19380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 761.186742] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.665954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.671759] Lustre: Skipped 1 previous similar message [ 761.744399] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 766.976624] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 766.993975] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 767.009892] Lustre: Skipped 2 previous similar messages [ 767.015512] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 767.117486] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 767.182131] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:106 to 0x2c0000401:161) [ 767.182443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:161) [ 779.695809] Lustre: DEBUG MARKER: == sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 17:33:55 (1788212035) [ 796.057340] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 814.066728] Lustre: 28503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 837.715826] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 841.564522] Lustre: 29445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 858.103341] Lustre: *** cfs_fail_loc=198, val=0*** [ 871.177227] Lustre: Failing over lustre-MDT0000 [ 871.600416] Lustre: server umount lustre-MDT0000 complete [ 874.472338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.481605] LustreError: 24784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 874.491534] Lustre: Skipped 1 previous similar message [ 874.516464] LustreError: 24784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 876.067385] LustreError: 23870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788212133 with bad export cookie 7899596791058502101 [ 876.070509] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 876.073332] Lustre: Failing over lustre-MDT0001 [ 876.077403] LustreError: 23870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 876.502543] Lustre: server umount lustre-MDT0001 complete [ 887.255530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 887.316608] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 893.007662] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 894.477916] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 894.490702] Lustre: Skipped 3 previous similar messages [ 895.775179] Lustre: 14734:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788212136/real 1788212136] req@ffff99102fadb100 x1875075817449984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788212152 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 895.813742] Lustre: 14734:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 902.632952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 902.995872] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:193) [ 902.999995] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:193) [ 907.904494] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 908.260186] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 908.362973] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 908.418959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:202 to 0x280000401:225) [ 908.419903] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:202 to 0x2c0000401:225) [ 910.899822] Lustre: lustre-MDT0001: trigger partial OI scrub for RPC inconsistency, checking FID [0x2400013a1:0x52:0x0]/0x31a): rc = 0 [ 920.286879] Lustre: DEBUG MARKER: == sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 17:36:16 (1788212176) [ 922.139850] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_1c ldiskfs special test [ 924.557347] Lustre: DEBUG MARKER: == sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 17:36:20 (1788212180) [ 926.771727] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_2 ldiskfs special test [ 929.580898] Lustre: DEBUG MARKER: == sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 17:36:24 (1788212184) [ 948.093283] Lustre: *** cfs_fail_loc=193, val=0*** [ 948.108718] Lustre: Skipped 1 previous similar message [ 958.255029] Lustre: Failing over lustre-MDT0000 [ 958.655642] Lustre: server umount lustre-MDT0000 complete [ 959.456828] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.461106] LustreError: 32113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.474613] Lustre: Skipped 5 previous similar messages [ 959.479707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 959.492934] LustreError: 32113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 963.793098] LustreError: 23870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788212221 with bad export cookie 7899596791058529443 [ 963.802365] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 963.803220] Lustre: Failing over lustre-MDT0001 [ 963.810563] LustreError: 23870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 964.483312] Lustre: server umount lustre-MDT0001 complete [ 979.935137] Lustre: 14734:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788212221/real 1788212221] req@ffff99103d76ca80 x1875075817578880/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788212237 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 979.979248] Lustre: 14734:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 989.217063] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6da1016d77dc692a [ 989.233753] Lustre: MGC192.168.202.119@tcp: Connection restored to 0@lo (at 0@lo) [ 989.245232] Lustre: Skipped 4 previous similar messages [ 990.051888] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 993.758917] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1001.949495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1002.030316] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1002.327923] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:233 to 0x280000400:257) [ 1002.331295] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:234 to 0x2c0000400:257) [ 1007.185661] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1007.785688] Lustre: lustre-MDT0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 1007.814706] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:265 to 0x2c0000401:289) [ 1007.821224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:266 to 0x280000401:289) [ 1012.043353] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200002342:0x4e:0x0]/0x3f2): rc = 0 [ 1012.085156] LustreError: 34321:0:(osd_index.c:203:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -17 [ 1035.130278] Lustre: DEBUG MARKER: == sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 17:38:10 (1788212290) [ 1037.108744] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_4b ldiskfs special test [ 1038.797315] Lustre: DEBUG MARKER: == sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ========================================================== 17:38:15 (1788212295) [ 1040.496730] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_4c ldiskfs special test [ 1042.413583] Lustre: DEBUG MARKER: == sanity-scrub test 4d: FID in LMA mismatch with object FID won't block create ========================================================== 17:38:18 (1788212298) [ 1043.749996] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_4d ldiskfs only test [ 1045.425892] Lustre: DEBUG MARKER: == sanity-scrub test 4e: FID reuse can be fixed ========== 17:38:21 (1788212301) [ 1047.267323] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_4e ldiskfs only test [ 1048.601972] Lustre: DEBUG MARKER: == sanity-scrub test 5: OI scrub state machine =========== 17:38:25 (1788212305) [ 1063.904146] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.904954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1063.933077] Lustre: Skipped 5 previous similar messages [ 1063.949661] Lustre: Skipped 3 previous similar messages [ 1066.052985] Lustre: server umount lustre-MDT0000 complete [ 1069.499324] LustreError: 32115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788212326 with bad export cookie 7899596791058557226 [ 1069.507099] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1069.511081] LustreError: 32115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1074.148164] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1074.153321] Lustre: Skipped 1 previous similar message [ 1076.037831] Lustre: server umount lustre-MDT0001 complete [ 1086.736535] Lustre: server umount lustre-OST0000 complete [ 1097.472534] Lustre: server umount lustre-OST0001 complete [ 1107.693345] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 1118.543503] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1131.864673] vdc: vdc1 vdc9 [ 1131.877762] vdc: vdc1 vdc9 [ 1131.901983] vdc: vdc1 vdc9 [ 1142.951444] vdd: vdd1 vdd9 [ 1142.985274] vdd: vdd1 vdd9 [ 1143.029793] vdd: vdd1 vdd9 [ 1154.282518] vde: vde1 vde9 [ 1154.339240] vde: vde1 vde9 [ 1154.371479] vde: vde1 vde9 [ 1165.372430] vdf: vdf1 vdf9 [ 1185.299583] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1195.432213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1195.494392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1195.569934] Lustre: lustre-MDT0000: new disk, initializing [ 1195.826566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.830447] Lustre: Skipped 3 previous similar messages [ 1195.875265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1200.435488] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1211.471417] Lustre: 39522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1211.582592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1211.586764] Lustre: Skipped 1 previous similar message [ 1211.661872] Lustre: lustre-MDT0001: new disk, initializing [ 1212.004187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1212.014983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1216.777202] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1222.186431] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1230.242267] Lustre: lustre-OST0000: new disk, initializing [ 1230.245166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1231.574455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1231.587509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1231.801158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1238.131524] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1253.146712] Lustre: lustre-OST0001: new disk, initializing [ 1253.149773] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1254.533653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1254.545780] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1254.859055] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1260.830457] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1271.581596] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1275.273928] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1289.389878] Lustre: *** cfs_fail_loc=193, val=0*** [ 1289.392953] Lustre: Skipped 7 previous similar messages [ 1300.677528] Lustre: Failing over lustre-MDT0000 [ 1301.264503] Lustre: server umount lustre-MDT0000 complete [ 1304.042162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1304.050920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1304.071013] Lustre: Skipped 3 previous similar messages [ 1304.082662] LustreError: 39532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.112695] LustreError: 39532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 1305.638564] LustreError: 41192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788212562 with bad export cookie 7899596791058559067 [ 1305.644984] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1305.649335] Lustre: Failing over lustre-MDT0001 [ 1305.651782] LustreError: 41192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1306.121665] Lustre: server umount lustre-MDT0001 complete [ 1314.911192] LustreError: 43887:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1314.929077] LustreError: 43887:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9910382fd880 x1875075817755264/t0(0) o250->MGC192.168.202.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788212572 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:4294967295 [ 1314.972169] LustreError: 43887:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1315.816568] LustreError: 14730:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9910382fd880 x1875075817756416/t0(0) o250->MGC192.168.202.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1316.566599] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1320.675953] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1325.549668] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1325.554706] Lustre: Skipped 5 previous similar messages [ 1326.560103] Lustre: 14732:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788212567/real 1788212567] req@ffff991008848000 x1875075817755776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788212583 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1326.593341] Lustre: 14732:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 1329.164930] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:65) [ 1329.166659] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:65) [ 1334.350431] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1334.420272] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1334.454171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 1334.457344] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:65) [ 1334.547812] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1341.341157] Lustre: *** cfs_fail_loc=190, val=3*** [ 1341.341464] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000402:0x4e:0x0]/0x23a): rc = 0 [ 1341.350235] Lustre: Skipped 1 previous similar message [ 1341.352889] LustreError: 45249:0:(osd_index.c:203:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -17 [ 1341.366201] LustreError: 45249:0:(osd_index.c:203:__osd_xattr_load_by_oid()) Skipped 3 previous similar messages [ 1344.418729] Lustre: *** cfs_fail_loc=190, val=3*** [ 1344.421301] Lustre: Skipped 3 previous similar messages [ 1347.423501] Lustre: *** cfs_fail_loc=191, val=3*** [ 1347.434865] Lustre: Skipped 1 previous similar message [ 1352.747252] Lustre: Failing over lustre-MDT0000 [ 1353.166734] Lustre: server umount lustre-MDT0000 complete [ 1356.902553] Lustre: Failing over lustre-MDT0001 [ 1356.906444] LustreError: 39512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788212614 with bad export cookie 7899596791058578086 [ 1356.906841] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1356.912645] LustreError: 39512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1357.309063] Lustre: server umount lustre-MDT0001 complete [ 1382.368754] LustreError: 14730:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff99100884d880 x1875075817791744/t0(0) o250->MGC192.168.202.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1383.039748] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1388.428486] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1397.951067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1397.965352] LustreError: Skipped 1 previous similar message [ 1398.037039] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1398.363242] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:97) [ 1398.372066] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:97) [ 1402.887827] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1403.618060] Lustre: lustre-MDT0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 1403.666012] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:97) [ 1403.670038] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:97) [ 1409.433354] Lustre: Failing over lustre-MDT0000 [ 1409.851201] Lustre: server umount lustre-MDT0000 complete [ 1414.721890] LustreError: 41192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788212671 with bad export cookie 7899596791058579794 [ 1414.729615] Lustre: Failing over lustre-MDT0001 [ 1415.062411] Lustre: server umount lustre-MDT0001 complete [ 1424.271752] Lustre: *** cfs_fail_loc=190, val=3*** [ 1424.273076] Lustre: Skipped 1 previous similar message [ 1430.367418] Lustre: *** cfs_fail_loc=190, val=3*** [ 1430.373867] Lustre: Skipped 1 previous similar message [ 1439.455799] Lustre: *** cfs_fail_loc=190, val=3*** [ 1439.457844] Lustre: Skipped 2 previous similar messages [ 1439.534285] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1444.139559] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1452.489898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1452.493737] Lustre: Skipped 8 previous similar messages [ 1452.558155] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:129) [ 1452.560975] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:129) [ 1453.479497] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1453.633209] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1453.633318] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1453.643718] Lustre: Skipped 13 previous similar messages [ 1453.686163] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:129) [ 1453.690565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:129) [ 1457.088925] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1457.503502] Lustre: *** cfs_fail_loc=190, val=3*** [ 1457.510834] Lustre: Skipped 7 previous similar messages [ 1466.658877] LustreError: 49799:0:(osd_index.c:203:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -17 [ 1466.659784] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000402:0x45:0x0]/0x1f5): rc = 0 [ 1466.667927] LustreError: 49799:0:(osd_index.c:203:__osd_xattr_load_by_oid()) Skipped 1 previous similar message [ 1466.707534] Lustre: Skipped 1 previous similar message [ 1475.978318] Lustre: DEBUG MARKER: == sanity-scrub test 6: OI scrub resumes from last checkpoint ========================================================== 17:45:32 (1788212732) [ 1495.877504] Lustre: *** cfs_fail_loc=193, val=0*** [ 1495.879306] Lustre: Skipped 7 previous similar messages [ 1510.314811] Lustre: Failing over lustre-MDT0000 [ 1510.840298] Lustre: server umount lustre-MDT0000 complete [ 1514.980720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1514.990377] LustreError: Skipped 1 previous similar message [ 1515.822625] LustreError: 39512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788212773 with bad export cookie 7899596791058581425 [ 1515.823196] Lustre: Failing over lustre-MDT0001 [ 1515.824055] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1515.824062] LustreError: Skipped 1 previous similar message [ 1515.833870] LustreError: 39512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1516.511721] Lustre: server umount lustre-MDT0001 complete [ 1536.479195] Lustre: 14732:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788212777/real 1788212777] req@ffff99103c33f800 x1875075817943552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788212793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1536.505291] Lustre: 14732:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 26 previous similar messages [ 1541.663536] LustreError: 14730:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff99103c33ca80 x1875075817945344/t0(0) o250->MGC192.168.202.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1542.416987] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1547.203524] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1555.708490] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:169 to 0x2c0000400:193) [ 1555.710388] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:193) [ 1561.085506] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1561.307132] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1561.355096] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:170 to 0x2c0000401:193) [ 1561.355884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 1561.387456] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1567.960574] Lustre: *** cfs_fail_loc=190, val=2*** [ 1567.962510] Lustre: Skipped 5 previous similar messages [ 1567.964992] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200001b72:0x4e:0x0]/0x3a0): rc = 0 [ 1567.992217] Lustre: Skipped 1 previous similar message [ 1568.012430] LustreError: 52489:0:(osd_index.c:203:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -17 [ 1568.024781] LustreError: 52489:0:(osd_index.c:203:__osd_xattr_load_by_oid()) Skipped 1 previous similar message [ 1579.906409] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200001b72:0x45:0x0]/0x31a): rc = 0 [ 1579.915832] Lustre: Skipped 1 previous similar message [ 1579.922340] LustreError: 52738:0:(osd_index.c:203:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -17 [ 1579.936168] LustreError: 52738:0:(osd_index.c:203:__osd_xattr_load_by_oid()) Skipped 1 previous similar message [ 1593.623556] Lustre: Failing over lustre-MDT0000 [ 1593.922399] Lustre: server umount lustre-MDT0000 complete [ 1596.898988] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.919920] LustreError: 51236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1596.932053] Lustre: Skipped 24 previous similar messages [ 1596.963653] LustreError: 51236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 52 previous similar messages [ 1598.457273] Lustre: Failing over lustre-MDT0001 [ 1599.031946] Lustre: server umount lustre-MDT0001 complete [ 1630.226468] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1632.674528] Lustre: *** cfs_fail_loc=190, val=3*** [ 1632.676395] Lustre: Skipped 27 previous similar messages [ 1638.523136] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:225) [ 1638.527847] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:169 to 0x2c0000400:225) [ 1642.787487] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1643.723270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:225) [ 1643.726887] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:170 to 0x2c0000401:225) [ 1664.077782] Lustre: DEBUG MARKER: == sanity-scrub test 7: System is available during OI scrub scanning ========================================================== 17:48:39 (1788212919) [ 1682.115716] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1699.443749] Lustre: 56886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1721.599215] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1726.524953] Lustre: 57832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1760.501681] Lustre: *** cfs_fail_loc=193, val=0*** [ 1760.503850] Lustre: Skipped 7 previous similar messages [ 1771.141561] Lustre: Failing over lustre-MDT0000 [ 1771.489394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1771.495635] Lustre: Skipped 3 previous similar messages [ 1771.538208] Lustre: server umount lustre-MDT0000 complete [ 1775.535167] LustreError: 39513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788213032 with bad export cookie 7899596791058624748 [ 1775.539251] Lustre: Failing over lustre-MDT0001 [ 1775.542606] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1775.542615] LustreError: Skipped 1 previous similar message [ 1775.543861] LustreError: 39513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 9 previous similar messages [ 1775.920155] Lustre: server umount lustre-MDT0001 complete [ 1795.999424] Lustre: 14731:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788213037/real 1788213037] req@ffff990f109e2a00 x1875075818155904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788213053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1796.040242] Lustre: 14731:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 24 previous similar messages [ 1800.898148] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1800.908440] Lustre: Skipped 1 previous similar message [ 1805.368680] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1811.430224] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1811.435576] Lustre: Skipped 10 previous similar messages [ 1815.972410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1815.988645] LustreError: Skipped 2 previous similar messages [ 1816.040869] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1816.051665] Lustre: Skipped 1 previous similar message [ 1816.442600] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:265 to 0x2c0000400:289) [ 1816.453589] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:266 to 0x280000400:289) [ 1821.758625] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1821.823406] Lustre: lustre-MDT0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 1821.834487] Lustre: Skipped 1 previous similar message [ 1821.883503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 1821.887529] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:266 to 0x2c0000401:289) [ 1828.520593] Lustre: *** cfs_fail_loc=190, val=3*** [ 1828.522486] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200002b12:0x4e:0x0]/0x334): rc = 0 [ 1828.527580] Lustre: Skipped 9 previous similar messages [ 1828.562255] Lustre: Skipped 1 previous similar message [ 1828.585252] LustreError: 60428:0:(osd_index.c:203:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -17 [ 1828.606381] LustreError: 60428:0:(osd_index.c:203:__osd_xattr_load_by_oid()) Skipped 1 previous similar message [ 1853.603323] Lustre: DEBUG MARKER: == sanity-scrub test 8: Control OI scrub manually ======== 17:51:49 (1788213109) [ 1890.917101] Lustre: *** cfs_fail_loc=193, val=0*** [ 1890.921363] Lustre: Skipped 7 previous similar messages [ 1901.381446] Lustre: Failing over lustre-MDT0000 [ 1901.674144] Lustre: server umount lustre-MDT0000 complete [ 1906.026419] Lustre: Failing over lustre-MDT0001 [ 1906.417572] Lustre: server umount lustre-MDT0001 complete [ 1915.815573] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6da1016d77df976f [ 1920.717956] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1929.834639] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:330 to 0x280000400:353) [ 1929.845544] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:329 to 0x2c0000400:353) [ 1934.727406] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1935.016784] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:330 to 0x2c0000401:353) [ 1935.016996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:329 to 0x280000401:353) [ 1956.590273] LustreError: 64189:0:(osd_index.c:203:__osd_xattr_load_by_oid()) lustre-MDT0001: can't get bonus, rc = -17 [ 1956.601213] LustreError: 64189:0:(osd_index.c:203:__osd_xattr_load_by_oid()) Skipped 3 previous similar messages [ 1969.292475] Lustre: DEBUG MARKER: == sanity-scrub test 9: OI scrub speed control =========== 17:53:44 (1788213224) [ 1971.055348] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_9 test scrub speed only on ldiskfs [ 1973.206851] Lustre: DEBUG MARKER: == sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount (1) ========================================================== 17:53:48 (1788213228) [ 1992.525882] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2009.198347] Lustre: 66107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2030.508695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2052.506187] Lustre: *** cfs_fail_loc=193, val=0*** [ 2052.515632] Lustre: Skipped 7 previous similar messages [ 2064.067407] Lustre: Failing over lustre-MDT0000 [ 2064.264856] Lustre: server umount lustre-MDT0000 complete [ 2067.855204] LustreError: 39513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788213325 with bad export cookie 7899596791058765679 [ 2067.857332] Lustre: Failing over lustre-MDT0001 [ 2067.858238] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2067.858244] LustreError: Skipped 1 previous similar message [ 2067.868737] LustreError: 39513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2067.955470] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2068.399313] Lustre: server umount lustre-MDT0001 complete [ 2094.035158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.039145] Lustre: Skipped 8 previous similar messages [ 2094.116210] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2094.119325] Lustre: Skipped 1 previous similar message [ 2098.802194] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2108.963720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2108.969915] LustreError: Skipped 1 previous similar message [ 2109.017111] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2109.025100] Lustre: Skipped 1 previous similar message [ 2109.396465] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:394 to 0x2c0000400:417) [ 2109.402957] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:393 to 0x280000400:417) [ 2114.732843] Lustre: lustre-MDT0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 2114.736928] Lustre: Skipped 1 previous similar message [ 2114.770964] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:394 to 0x2c0000401:417) [ 2114.772517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:393 to 0x280000401:417) [ 2114.889872] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2120.770127] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200003ab2:0x4e:0x0]/0x4a6): rc = 0 [ 2120.771828] Lustre: *** cfs_fail_loc=190, val=1*** [ 2120.797304] Lustre: Skipped 1 previous similar message [ 2120.814825] Lustre: Skipped 31 previous similar messages [ 2120.820039] LustreError: 69333:0:(osd_index.c:203:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -17 [ 2120.834485] LustreError: 69333:0:(osd_index.c:203:__osd_xattr_load_by_oid()) Skipped 3 previous similar messages [ 2129.416806] Lustre: Failing over lustre-MDT0000 [ 2129.780910] Lustre: server umount lustre-MDT0000 complete [ 2129.890276] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.894916] Lustre: Skipped 20 previous similar messages [ 2129.895713] LustreError: 41184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.907478] LustreError: 41184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 48 previous similar messages [ 2133.538491] Lustre: Failing over lustre-MDT0001 [ 2133.899941] Lustre: server umount lustre-MDT0001 complete [ 2164.691786] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2175.389450] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:393 to 0x280000400:449) [ 2175.391365] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:394 to 0x2c0000400:449) [ 2180.213566] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2180.775561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:394 to 0x2c0000401:449) [ 2180.779357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:393 to 0x280000401:449) [ 2185.429710] Lustre: Failing over lustre-MDT0000 [ 2185.701734] Lustre: server umount lustre-MDT0000 complete [ 2189.252458] Lustre: Failing over lustre-MDT0001 [ 2189.555647] Lustre: server umount lustre-MDT0001 complete [ 2203.985598] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2213.006982] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:393 to 0x280000400:481) [ 2213.010955] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:394 to 0x2c0000400:481) [ 2217.717098] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2218.214427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:393 to 0x280000401:481) [ 2218.216146] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:394 to 0x2c0000401:481) [ 2223.499432] LustreError: 72225:0:(osd_index.c:203:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -17 [ 2223.509954] LustreError: 72225:0:(osd_index.c:203:__osd_xattr_load_by_oid()) Skipped 1 previous similar message [ 2235.493994] Lustre: DEBUG MARKER: == sanity-scrub test 11: OI scrub skips the new created objects only once ========================================================== 17:58:11 (1788213491) [ 2238.191408] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_11 ldiskfs special test [ 2240.071722] Lustre: DEBUG MARKER: == sanity-scrub test 12: OI scrub can rebuild invalid /O entries ========================================================== 17:58:16 (1788213496) [ 2257.166200] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2272.895662] Lustre: 75231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2272.903612] Lustre: 75231:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2296.167251] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2306.720479] Lustre: *** cfs_fail_loc=195, val=0*** [ 2306.722357] Lustre: Skipped 7 previous similar messages [ 2312.050544] Lustre: Failing over lustre-OST0000 [ 2312.298780] Lustre: server umount lustre-OST0000 complete [ 2327.807889] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2330.640197] LustreError: 57855:0:(osd_object.c:462:osd_check_lma()) lustre-OST0000: FID-in-LMA [0x280000401:0x20c:0x0] does not match the object self-fid [0x280000401:0x20b:0x0]: rc = -78 [ 2507.920750] LustreError: 50212:0:(osd_object.c:462:osd_check_lma()) lustre-OST0000: FID-in-LMA [0x280000401:0x213:0x0] does not match the object self-fid [0x280000401:0x212:0x0]: rc = -78 [ 2507.932444] LustreError: 50212:0:(osd_object.c:462:osd_check_lma()) Skipped 8 previous similar messages [ 2508.942261] LustreError: 50224:0:(osd_object.c:462:osd_check_lma()) lustre-OST0000: FID-in-LMA [0x280000401:0x214:0x0] does not match the object self-fid [0x280000401:0x213:0x0]: rc = -78 [ 2508.961350] LustreError: 50224:0:(osd_object.c:462:osd_check_lma()) Skipped 19 previous similar messages [ 2510.960497] LustreError: 77540:0:(osd_index.c:203:__osd_xattr_load_by_oid()) lustre-OST0000: can't get bonus, rc = -2 [ 2510.977560] LustreError: 77540:0:(osd_index.c:203:__osd_xattr_load_by_oid()) Skipped 1 previous similar message [ 2521.438499] Lustre: DEBUG MARKER: == sanity-scrub test 13: OI scrub can rebuild missed /O entries ========================================================== 18:02:57 (1788213777) [ 2526.632775] Lustre: *** cfs_fail_loc=196, val=0*** [ 2526.641448] Lustre: Skipped 63 previous similar messages [ 2532.596103] Lustre: Failing over lustre-OST0000 [ 2532.814897] Lustre: server umount lustre-OST0000 complete [ 2543.565447] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2543.577811] Lustre: Skipped 27 previous similar messages [ 2550.151079] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2742.579957] Lustre: DEBUG MARKER: == sanity-scrub test 14: OI scrub can repair OST objects under lost+found ========================================================== 18:06:38 (1788213998) [ 2744.233833] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_14 ldiskfs special test [ 2746.331339] Lustre: DEBUG MARKER: == sanity-scrub test 15: Dryrun mode OI scrub ============ 18:06:42 (1788214002) [ 2762.721399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.728560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.755147] Lustre: Skipped 16 previous similar messages [ 2762.771596] Lustre: Skipped 4 previous similar messages [ 2765.024536] Lustre: server umount lustre-MDT0000 complete [ 2767.841798] LustreError: 72237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.856048] LustreError: 72237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 2769.376141] LustreError: 50209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788214026 with bad export cookie 7899596791058811053 [ 2769.378744] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2769.388235] LustreError: 50209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 12 previous similar messages [ 2769.416130] LustreError: Skipped 2 previous similar messages [ 2771.843222] Lustre: server umount lustre-MDT0001 complete [ 2789.343274] Lustre: 14734:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788214030/real 1788214030] req@ffff991006fa6680 x1875075818858496/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788214046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2789.372348] Lustre: 14734:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 36 previous similar messages [ 2793.666255] Lustre: server umount lustre-OST0000 complete [ 2798.008400] Lustre: server umount lustre-OST0001 complete [ 2806.982685] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 2816.149665] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2828.306872] vdc: vdc1 vdc9 [ 2828.358309] vdc: vdc1 vdc9 [ 2840.477689] vdd: vdd1 vdd9 [ 2852.485219] vde: vde1 vde9 [ 2865.395699] vdf: vdf1 vdf9 [ 2888.020847] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2900.100977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2900.217243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2900.325879] Lustre: lustre-MDT0000: new disk, initializing [ 2900.803758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2900.809594] Lustre: Skipped 7 previous similar messages [ 2900.879495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2906.464342] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2920.574539] Lustre: 83722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2920.596492] Lustre: 83722:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2920.745405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2920.765885] Lustre: Skipped 1 previous similar message [ 2920.896097] Lustre: lustre-MDT0001: new disk, initializing [ 2921.349845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2921.362142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2927.673416] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2933.626829] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2941.153281] Lustre: lustre-OST0000: new disk, initializing [ 2941.158376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2942.652629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2942.664576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2942.919943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2949.379536] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2964.773903] Lustre: lustre-OST0001: new disk, initializing [ 2964.782296] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2966.636315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2966.662873] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2967.053535] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2973.256509] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2983.567234] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2987.882715] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3001.075901] Lustre: *** cfs_fail_loc=193, val=0*** [ 3001.081764] Lustre: Skipped 63 previous similar messages [ 3013.780299] Lustre: Failing over lustre-MDT0000 [ 3014.326633] Lustre: server umount lustre-MDT0000 complete [ 3020.060102] Lustre: Failing over lustre-MDT0001 [ 3020.669100] Lustre: server umount lustre-MDT0001 complete [ 3045.520356] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3045.534737] Lustre: Skipped 4 previous similar messages [ 3050.967879] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3060.963947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 3060.978825] LustreError: Skipped 5 previous similar messages [ 3061.014659] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3061.023555] Lustre: Skipped 4 previous similar messages [ 3061.258370] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:42 to 0x2c0000400:65) [ 3061.258665] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:65) [ 3066.467405] Lustre: lustre-MDT0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 3066.482748] Lustre: Skipped 4 previous similar messages [ 3066.527060] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:65) [ 3066.531617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3066.570098] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3072.625255] LustreError: 89381:0:(osd_index.c:203:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -17 [ 3072.640890] LustreError: 89381:0:(osd_index.c:203:__osd_xattr_load_by_oid()) Skipped 2 previous similar messages [ 3122.311538] Lustre: DEBUG MARKER: == sanity-scrub test 16: Initial OI scrub can rebuild crashed index objects ========================================================== 18:12:57 (1788214377) [ 3138.369885] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3156.252781] Lustre: 92721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3176.918818] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3209.671112] Lustre: Failing over lustre-MDT0000 [ 3209.701014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3209.708733] Lustre: *** cfs_fail_loc=199, val=0*** [ 3209.716357] Lustre: Skipped 1 previous similar message [ 3209.724137] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000001:0x3:0x0]: rc = 0 [ 3209.734785] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x3:0x0]: rc = 0 [ 3209.740627] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xa:0x0]: rc = 0 [ 3209.746899] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xb:0x0]: rc = 0 [ 3209.753746] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xc:0x0]: rc = 0 [ 3209.763494] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xd:0x0]: rc = 0 [ 3209.769265] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xe:0x0]: rc = 0 [ 3209.777837] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xf:0x0]: rc = 0 [ 3209.782856] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x11:0x0]: rc = 0 [ 3209.789110] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x36:0x0]: rc = 0 [ 3209.795063] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x1:0x0]: rc = 0 [ 3209.800815] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x2:0x0]: rc = 0 [ 3209.805785] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x3:0x0]: rc = 0 [ 3209.811697] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20001:0x0]: rc = 0 [ 3209.817964] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20002:0x0]: rc = 0 [ 3209.823226] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20003:0x0]: rc = 0 [ 3209.829899] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x10000:0x0]: rc = 0 [ 3209.836274] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x20000:0x0]: rc = 0 [ 3209.849187] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x1010000:0x0]: rc = 0 [ 3209.856570] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x1020000:0x0]: rc = 0 [ 3209.862914] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x2010000:0x0]: rc = 0 [ 3209.871587] Lustre: 94029:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x2020000:0x0]: rc = 0 [ 3210.104389] Lustre: server umount lustre-MDT0000 complete [ 3213.832348] Lustre: Failing over lustre-MDT0001 [ 3213.851399] Lustre: *** cfs_fail_loc=199, val=0*** [ 3213.853706] Lustre: Skipped 20 previous similar messages [ 3213.856452] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000001:0x3:0x0]: rc = 0 [ 3213.862406] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x4:0x0]: rc = 0 [ 3213.868515] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x5:0x0]: rc = 0 [ 3213.874182] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x6:0x0]: rc = 0 [ 3213.880511] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x7:0x0]: rc = 0 [ 3213.890245] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x8:0x0]: rc = 0 [ 3213.896992] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x9:0x0]: rc = 0 [ 3213.902773] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xb:0x0]: rc = 0 [ 3213.906572] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x30:0x0]: rc = 0 [ 3213.912401] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x1:0x0]: rc = 0 [ 3213.918503] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x2:0x0]: rc = 0 [ 3213.924557] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x3:0x0]: rc = 0 [ 3213.931533] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20001:0x0]: rc = 0 [ 3213.935256] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20002:0x0]: rc = 0 [ 3213.941401] Lustre: 94232:0:(scrub.c:1092:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20003:0x0]: rc = 0 [ 3214.246551] Lustre: server umount lustre-MDT0001 complete [ 3224.292334] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3224.322342] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x36:0x0]: rc = 0 [ 3224.334886] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3224.354775] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x2020000-MDT0000' with [0x200000005:0x20003:0x0]: rc = 0 [ 3224.377115] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3224.393179] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x20000-MDT0000' with [0x200000005:0x20001:0x0]: rc = 0 [ 3224.417107] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3224.428156] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3224.440228] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000003:0xd:0x0]: rc = 0 [ 3224.457076] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3224.466904] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x1020000-MDT0000' with [0x200000005:0x20002:0x0]: rc = 0 [ 3224.479977] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000003:0xe:0x0]: rc = 0 [ 3224.486787] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3224.501319] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000003:0xf:0x0]: rc = 0 [ 3224.514383] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3224.533421] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3224.544629] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3224.561927] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3224.574021] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3224.589517] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3224.602379] Lustre: 94743:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3244.981607] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3250.157348] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3250.163466] Lustre: Skipped 6 previous similar messages [ 3254.205780] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3254.231171] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index 'nodemap' with [0x200000003:0x30:0x0]: rc = 0 [ 3254.242628] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index '0x20000' with [0x200000003:0x7:0x0]: rc = 0 [ 3254.250149] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3254.259821] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3254.267815] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index '0x1020000-MDT0001' with [0x200000005:0x20002:0x0]: rc = 0 [ 3254.275202] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3254.288509] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3254.304917] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index '0x2020000' with [0x200000003:0x9:0x0]: rc = 0 [ 3254.313542] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index '0x20000-MDT0001' with [0x200000005:0x20001:0x0]: rc = 0 [ 3254.320623] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3254.328973] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index '0x2020000-MDT0001' with [0x200000005:0x20003:0x0]: rc = 0 [ 3254.354052] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3254.379313] Lustre: 95514:0:(osd_scrub.c:952:osd_index_restore()) lustre-MDT0001: restore index '0x1020000' with [0x200000003:0x8:0x0]: rc = 0 [ 3255.337496] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:105 to 0x280000400:129) [ 3255.339283] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:106 to 0x2c0000400:129) [ 3260.547808] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3260.639824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 3260.659896] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:106 to 0x2c0000401:129) [ 3274.343714] Lustre: DEBUG MARKER: == sanity-scrub test 17a: ENOSPC on OI insert shouldn't leak inodes ========================================================== 18:15:29 (1788214529) [ 3276.201557] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_17a ldiskfs only test [ 3278.692821] Lustre: DEBUG MARKER: == sanity-scrub test 17b: ENOSPC on .. insertion shouldn't leak inodes ========================================================== 18:15:34 (1788214534) [ 3280.207275] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_17b ldiskfs only test [ 3282.600900] Lustre: DEBUG MARKER: == sanity-scrub test 18: test mount -o resetoi to recreate OI files ========================================================== 18:15:38 (1788214538) [ 3310.791458] Lustre: Failing over lustre-MDT0000 [ 3311.172043] Lustre: server umount lustre-MDT0000 complete [ 3315.876333] Lustre: Failing over lustre-MDT0001 [ 3316.381465] Lustre: server umount lustre-MDT0001 complete [ 3331.395401] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3341.791944] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3341.808288] Lustre: Skipped 4 previous similar messages [ 3342.251654] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:169 to 0x2c0000400:193) [ 3342.253293] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:193) [ 3347.693030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 3347.697514] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:170 to 0x2c0000401:193) [ 3348.450146] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3355.239140] Lustre: Failing over lustre-MDT0000 [ 3355.795824] Lustre: server umount lustre-MDT0000 complete [ 3360.104627] Lustre: Failing over lustre-MDT0001 [ 3360.406392] Lustre: server umount lustre-MDT0001 complete [ 3369.318831] Lustre: lustre-MDT0000: reset Object Index mappings [ 3370.353440] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3370.355447] LustreError: 93694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3370.364224] Lustre: Skipped 23 previous similar messages [ 3370.375031] LustreError: 93694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 3375.461832] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3386.312460] Lustre: lustre-MDT0001: reset Object Index mappings [ 3387.894495] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:225) [ 3387.906032] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:169 to 0x2c0000400:225) [ 3392.693195] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3393.309356] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:170 to 0x2c0000401:225) [ 3393.309716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:225) [ 3408.120580] Lustre: DEBUG MARKER: == sanity-scrub test 19: LFSCK can fix multiple linked files on OST ========================================================== 18:17:42 (1788214662) [ 3418.608248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3418.616528] Lustre: Skipped 3 previous similar messages [ 3423.714448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3423.733587] Lustre: Skipped 3 previous similar messages [ 3424.964380] Lustre: server umount lustre-MDT0000 complete [ 3429.526560] LustreError: 93761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788214686 with bad export cookie 7899596791058914044 [ 3429.532539] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3429.543141] LustreError: 93761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 18 previous similar messages [ 3429.557075] LustreError: Skipped 4 previous similar messages [ 3429.881324] Lustre: server umount lustre-MDT0001 complete [ 3444.394199] Lustre: server umount lustre-OST0000 complete [ 3459.410914] Lustre: server umount lustre-OST0001 complete [ 3499.487542] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3504.609812] LustreError: 102460:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 4: rc = -110 [ 3530.335504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3530.340235] Lustre: Skipped 11 previous similar messages [ 3537.301668] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3545.181141] Lustre: Failing over lustre-OST0000 [ 3545.390661] Lustre: server umount lustre-OST0000 complete [ 3586.847571] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3591.969801] LustreError: 104419:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 4: rc = -110 [ 3624.003636] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3632.472799] Lustre: DEBUG MARKER: == sanity-scrub test 20: Don't trigger OI scrub for irreparable oi repeatedly ========================================================== 18:21:28 (1788214888) [ 3634.440860] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_20 ldiskfs only test [ 3636.116064] Lustre: DEBUG MARKER: == sanity-scrub test 21: don't hang MDS recovery when failed to get update log ========================================================== 18:21:32 (1788214892) [ 3637.785361] Lustre: DEBUG MARKER: SKIP: sanity-scrub test_21 ldiskfs only test [ 3639.939869] Lustre: DEBUG MARKER: == sanity-scrub test 22: LFSCK can recreate or fix the LASTID on MDT/OST ========================================================== 18:21:35 (1788214895) [ 3649.487945] Lustre: server umount lustre-OST0000 complete [ 3681.369591] LustreError: 107116:0:(osd_index.c:203:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -2 [ 3681.376993] LustreError: 107116:0:(osd_index.c:203:__osd_xattr_load_by_oid()) Skipped 17 previous similar messages [ 3687.019459] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3693.244121] Lustre: Failing over lustre-MDT0000 [ 3693.303769] LustreError: 107174:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3693.310161] LustreError: 107174:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3693.315571] LustreError: 107174:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 3693.642663] Lustre: server umount lustre-MDT0000 complete [ 3730.736758] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3740.719680] Lustre: DEBUG MARKER: === sanity-scrub: start setup 18:23:16 (1788214996) === [ 3743.142604] LustreError: 109198:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3743.150530] LustreError: 109198:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3743.155925] LustreError: 109198:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 3743.474127] Lustre: server umount lustre-MDT0000 complete [ 3762.395580] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 3774.074620] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3789.061372] vdc: vdc1 vdc9 [ 3800.244130] vdd: vdd1 vdd9 [ 3813.143735] vde: vde1 vde9 [ 3829.656651] vdf: vdf1 vdf9 [ 3855.751892] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3867.269327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3867.747638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3867.870848] Lustre: lustre-MDT0000: new disk, initializing [ 3868.696831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3875.316918] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3887.892623] Lustre: 113740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3887.901673] Lustre: 113740:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 3887.975434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3887.978575] Lustre: Skipped 1 previous similar message [ 3888.097381] Lustre: lustre-MDT0001: new disk, initializing [ 3888.619692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3888.626836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3894.049977] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3899.887063] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3906.946078] Lustre: lustre-OST0000: new disk, initializing [ 3906.950548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3909.113748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3909.122310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3909.617783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3915.093135] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3926.299247] Lustre: lustre-OST0001: new disk, initializing [ 3926.303571] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3927.866859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3927.901452] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3928.350197] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3934.143410] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3945.366383] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3949.228488] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3954.816628] Lustre: DEBUG MARKER: === sanity-scrub: finish setup 18:26:50 (1788215210) === [ 3956.592743] Lustre: DEBUG MARKER: == sanity-scrub test complete, duration 3691 sec ========= 18:26:52 (1788215212) [ 3958.748524] Lustre: DEBUG MARKER: === sanity-scrub: start cleanup 18:26:54 (1788215214) === [ 3963.172740] Lustre: DEBUG MARKER: === sanity-scrub: finish cleanup 18:26:58 (1788215218) === [ 3969.010586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3969.024421] Lustre: Skipped 3 previous similar messages [ 3969.835871] Lustre: server umount lustre-MDT0000 complete [ 3974.120315] LustreError: 113747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3974.127602] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3974.148705] LustreError: 113747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages [ 3974.188567] Lustre: Skipped 9 previous similar messages [ 3974.362715] Lustre: server umount lustre-MDT0001 complete [ 3979.664286] Lustre: server umount lustre-OST0000 complete [ 3984.783789] Lustre: server umount lustre-OST0001 complete [ 3998.850439] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 4002.794568] Key type lgssc unregistered [ 4003.308665] LNet: 118676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4003.338298] LNetError: 118676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4003.362473] LNet: Removed LNI 192.168.202.119@tcp [ 4004.534164] Key type .llcrypt unregistered [ 4004.535830] Key type ._llcrypt unregistered