[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 467322280 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003254] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008604] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011117] LSM: Security Framework initializing [ 0.012108] Yama: becoming mindful. [ 0.013053] SELinux: Initializing. [ 0.014074] *** VALIDATE selinux *** [ 0.022339] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026800] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] *** VALIDATE tmpfs *** [ 0.030456] *** VALIDATE proc *** [ 0.031241] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.034263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041619] debug: unmapping init [mem 0xffffffff92859000-0xffffffff92860fff] [ 0.043189] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044740] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045026] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047015] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051018] ... event mask: 000000070000000f [ 0.052335] rcu: Hierarchical SRCU implementation. [ 0.054479] smp: Bringing up secondary CPUs ... [ 0.055628] x86: Booting SMP configuration: [ 0.056031] .... node #0, CPUs: #1 #2 #3 [ 0.060090] smp: Brought up 1 node, 4 CPUs [ 0.062019] smpboot: Max logical packages: 1 [ 0.063017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.254584] node 0 deferred pages initialised in 190ms [ 0.258254] devtmpfs: initialized [ 0.259235] x86/mm: Memory block size: 128MB [ 0.261924] gcov: version magic: 0x41383552 [ 0.263339] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.264083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.265244] pinctrl core: initialized pinctrl subsystem [ 0.266216] [ 0.266832] ************************************************************* [ 0.267016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.268016] ** ** [ 0.269015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.270014] ** ** [ 0.271019] ** This means that this kernel is built to expose internal ** [ 0.272015] ** IOMMU data structures, which may compromise security on ** [ 0.273017] ** your system. ** [ 0.274016] ** ** [ 0.275014] ** If you see this message and you are not debugging the ** [ 0.276012] ** kernel, report this immediately to your vendor! ** [ 0.277019] ** ** [ 0.278012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.279013] ************************************************************* [ 0.280815] NET: Registered protocol family 16 [ 0.281449] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.282069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.283067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.284436] cpuidle: using governor menu [ 0.286788] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.289572] PCI: Using configuration type 1 for base access [ 0.292131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.303074] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.306104] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.310160] cryptd: max_cpu_qlen set to 1000 [ 0.313347] ACPI: Added _OSI(Module Device) [ 0.316030] ACPI: Added _OSI(Processor Device) [ 0.317023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.319024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.325497] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.333533] ACPI: Interpreter enabled [ 0.335054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.337013] ACPI: Using IOAPIC for interrupt routing [ 0.339103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.342424] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.353204] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.356046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.358088] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.363102] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.368867] acpiphp: Slot [2] registered [ 0.371268] acpiphp: Slot [5] registered [ 0.372216] acpiphp: Slot [6] registered [ 0.374238] acpiphp: Slot [7] registered [ 0.376221] acpiphp: Slot [8] registered [ 0.378207] acpiphp: Slot [9] registered [ 0.380203] acpiphp: Slot [10] registered [ 0.382201] acpiphp: Slot [3] registered [ 0.384168] acpiphp: Slot [4] registered [ 0.385143] acpiphp: Slot [11] registered [ 0.387195] acpiphp: Slot [12] registered [ 0.389170] acpiphp: Slot [13] registered [ 0.390236] acpiphp: Slot [14] registered [ 0.392155] acpiphp: Slot [15] registered [ 0.394156] acpiphp: Slot [16] registered [ 0.396159] acpiphp: Slot [17] registered [ 0.398210] acpiphp: Slot [18] registered [ 0.400106] acpiphp: Slot [19] registered [ 0.402126] acpiphp: Slot [20] registered [ 0.403133] acpiphp: Slot [21] registered [ 0.404106] acpiphp: Slot [22] registered [ 0.406119] acpiphp: Slot [23] registered [ 0.407073] acpiphp: Slot [24] registered [ 0.408067] acpiphp: Slot [25] registered [ 0.409066] acpiphp: Slot [26] registered [ 0.410120] acpiphp: Slot [27] registered [ 0.413127] acpiphp: Slot [28] registered [ 0.415126] acpiphp: Slot [29] registered [ 0.416164] acpiphp: Slot [30] registered [ 0.418136] acpiphp: Slot [31] registered [ 0.420070] PCI host bridge to bus 0000:00 [ 0.421019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.423028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.427033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.429025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.431028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.433037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.436213] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.439260] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.443633] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.455796] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.461118] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.464051] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.466023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.470078] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.473275] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.475721] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.479054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.483667] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.489018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.502014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.506965] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.515202] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.524025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.531027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.554022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.565090] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.575021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.583022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.616019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.626523] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.634016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.643021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.672021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.683759] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.694040] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.704028] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.722022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.733725] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.744020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.750036] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.772026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.786080] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.797022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.804021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.821021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.833937] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.838157] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.840469] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.843421] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.845240] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.850153] iommu: Default domain type: Passthrough [ 0.853452] SCSI subsystem initialized [ 0.855106] ACPI: bus type USB registered [ 0.857140] usbcore: registered new interface driver usbfs [ 0.860157] usbcore: registered new interface driver hub [ 0.862074] usbcore: registered new device driver usb [ 0.863129] pps_core: LinuxPPS API ver. 1 registered [ 0.865017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.869069] PTP clock support registered [ 0.871066] EDAC MC: Ver: 3.0.0 [ 0.873194] PCI: Using ACPI for IRQ routing [ 0.876572] NetLabel: Initializing [ 0.877015] NetLabel: domain hash size = 128 [ 0.879024] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.881102] NetLabel: unlabeled traffic allowed by default [ 0.884220] vgaarb: loaded [ 0.887240] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.888014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.892868] clocksource: Switched to clocksource kvm-clock [ 0.996312] VFS: Disk quotas dquot_6.6.0 [ 0.998021] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.000720] *** VALIDATE ramfs *** [ 1.002133] *** VALIDATE hugetlbfs *** [ 1.003621] pnp: PnP ACPI init [ 1.005389] pnp: PnP ACPI: found 6 devices [ 1.025253] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.029820] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.032631] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.035750] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.039173] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.042287] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.045066] NET: Registered protocol family 2 [ 1.047608] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.053197] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.056583] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.061470] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.064860] TCP: Hash tables configured (established 65536 bind 65536) [ 1.067882] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.072206] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.075985] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.080173] NET: Registered protocol family 1 [ 1.085785] RPC: Registered named UNIX socket transport module. [ 1.088234] RPC: Registered udp transport module. [ 1.089945] RPC: Registered tcp transport module. [ 1.091902] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.094674] NET: Registered protocol family 44 [ 1.096428] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.098796] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.102065] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.104733] PCI: CLS 0 bytes, default 64 [ 1.106645] Unpacking initramfs... [ 2.481790] debug: unmapping init [mem 0xffff9d167cc54000-0xffff9d167ffbffff] [ 2.486025] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.488648] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.492143] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.973598] Initialise system trusted keyrings [ 2.975748] Key type blacklist registered [ 2.977615] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.986867] zbud: loaded [ 2.990138] *** VALIDATE nfs *** [ 2.991822] *** VALIDATE nfs4 *** [ 2.993605] pstore: using deflate compression [ 2.997969] Platform Keyring initialized [ 3.095134] NET: Registered protocol family 38 [ 3.096792] Key type asymmetric registered [ 3.098197] Asymmetric key parser 'x509' registered [ 3.099645] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.102640] io scheduler mq-deadline registered [ 3.104284] io scheduler kyber registered [ 3.106138] io scheduler bfq registered [ 3.108713] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.110804] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.113439] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.115938] ACPI: Power Button [PWRF] [ 3.121850] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.130164] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.145827] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.156423] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.178399] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.207147] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.236261] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.241656] Non-volatile memory driver v1.3 [ 3.243459] Linux agpgart interface v0.103 [ 3.276905] virtio_blk virtio1: [vda] 146752 512-byte logical blocks (75.1 MB/71.7 MiB) [ 3.280573] vda: detected capacity change from 0 to 75137024 [ 3.296363] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.299325] vdb: detected capacity change from 0 to 1073741824 [ 3.314911] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.317479] vdc: detected capacity change from 0 to 2621440000 [ 3.335895] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.339907] vdd: detected capacity change from 0 to 2621440000 [ 3.360180] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.363144] vde: detected capacity change from 0 to 4294967296 [ 3.378596] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.381709] vdf: detected capacity change from 0 to 4294967296 [ 3.388088] libphy: Fixed MDIO Bus: probed [ 3.393856] usbcore: registered new interface driver usbserial_generic [ 3.396781] usbserial: USB Serial support registered for generic [ 3.398973] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.403166] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.404555] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.406849] mousedev: PS/2 mouse device common for all mice [ 3.410873] rtc_cmos 00:05: RTC can wake from S4 [ 3.414188] rtc_cmos 00:05: registered as rtc0 [ 3.414588] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.415596] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.420550] intel_pstate: CPU model not supported [ 3.424041] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.426844] hid: raw HID events driver (C) Jiri Kosina [ 3.429336] usbcore: registered new interface driver usbhid [ 3.430855] usbhid: USB HID core driver [ 3.431023] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.432028] drop_monitor: Initializing network drop monitor service [ 3.435782] Initializing XFRM netlink socket [ 3.437036] NET: Registered protocol family 10 [ 3.439476] Segment Routing with IPv6 [ 3.440919] NET: Registered protocol family 17 [ 3.443492] mpls_gso: MPLS GSO support [ 3.447765] RAS: Correctable Errors collector initialized. [ 3.449557] AVX version of gcm_enc/dec engaged. [ 3.450848] AES CTR mode by8 optimization enabled [ 3.519325] sched_clock: Marking stable (3519283669, 0)->(4485761867, -966478198) [ 3.523556] registered taskstats version 1 [ 3.525730] Loading compiled-in X.509 certificates [ 3.528131] zswap: loaded using pool lzo/zbud [ 3.557101] Key type big_key registered [ 3.571088] Key type encrypted registered [ 3.573635] ima: No TPM chip found, activating TPM-bypass! [ 3.577130] ima: Allocated hash algorithm: sha1 [ 3.579475] ima: No architecture policies found [ 3.582075] evm: Initialising EVM extended attributes: [ 3.584823] evm: security.selinux [ 3.586485] evm: security.ima [ 3.588012] evm: security.capability [ 3.589702] evm: HMAC attrs: 0x1 [ 3.592529] rtc_cmos 00:05: setting system clock to 2026-09-08 03:20:39 UTC (1788837639) [ 3.598792] debug: unmapping init [mem 0xffffffff93803000-0xffffffff939fffff] [ 3.602025] debug: unmapping init [mem 0xffffffff92582000-0xffffffff92858fff] [ 3.611128] Write protecting the kernel read-only data: 28672k [ 3.615024] debug: unmapping init [mem 0xffffffff90c03000-0xffffffff90dfffff] [ 3.618148] debug: unmapping init [mem 0xffffffff91514000-0xffffffff915fffff] [ 3.651322] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.661918] systemd[1]: Detected virtualization kvm. [ 3.663982] systemd[1]: Detected architecture x86-64. [ 3.666232] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.696972] systemd[1]: No hostname configured. [ 3.698484] systemd[1]: Set hostname to . [ 3.700824] random: systemd: uninitialized urandom read (16 bytes read) [ 3.703765] systemd[1]: Initializing machine ID from random generator. [ 3.742465] random: ln: uninitialized urandom read (6 bytes read) [ 3.848833] random: systemd: uninitialized urandom read (16 bytes read) [ 3.852303] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.857266] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.863315] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.586111] device-mapper: uevent: version 1.0.3 [ 4.588767] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.387257] random: fast init done [ 5.391862] virtio_net virtio0 ens2: renamed from eth0 [ 5.409941] scsi host0: ata_piix [ 5.421870] scsi host1: ata_piix [ 5.423899] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.427702] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.968052] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.095720] random: crng init done [ 10.097049] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.627534] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.895887] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.154444] SELinux: Disabled at runtime. [ 12.217909] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.228496] systemd[1]: Detected virtualization kvm. [ 12.230898] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.767042] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.771151] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.775478] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.780207] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.785717] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.797453] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.803930] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ 12.923272] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.381882] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.151547] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.261863] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.830750] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.904761] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit)[ 19.361280] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (6s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit)[ 20.145332] NFS: Registering the id_resolver key type [ 20.156808] Key type id_resolver registered [ 20.163160] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg310-server login: [ 38.379080] hrtimer: interrupt took 4429804 ns [ 97.057924] libcfs: loading out-of-tree module taints kernel. [ 97.158085] Key type ._llcrypt registered [ 97.163309] Key type .llcrypt registered [ 97.323144] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 117.957928] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 119.675490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 119.720060] alg: No test for adler32 (adler32-zlib) [ 121.364888] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 122.422801] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 124.233875] Key type lgssc registered [ 126.465325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 147.099306] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 192.964404] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 208.238829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.284344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.682486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.733665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.871962] Lustre: lustre-MDT0000: new disk, initializing [ 210.006253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.045132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 214.740853] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.086720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.251220] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 228.309434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 228.316249] Lustre: Skipped 1 previous similar message [ 228.453951] Lustre: lustre-MDT0001: new disk, initializing [ 228.511241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.546893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 228.564224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.768889] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.742487] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 247.927379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.294464] Lustre: lustre-OST0000: new disk, initializing [ 248.298930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.305911] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.371884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.027523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 255.045511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 255.122875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 256.139945] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.335560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.464258] Lustre: lustre-OST0001: new disk, initializing [ 271.467236] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 271.472372] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.528810] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 277.075829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 277.083984] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 277.121680] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.394647] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 289.756365] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 301.266747] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 309.765475] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing check_logdir /tmp/testlogs/ [ 315.800767] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing yml_node [ 319.935358] Lustre: DEBUG MARKER: Client: 2.17.57.117 [ 323.472713] Lustre: DEBUG MARKER: MDS: 2.17.57.117 [ 326.022525] Lustre: DEBUG MARKER: OSS: 2.17.57.117 [ 327.580720] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Mon Sep 7 23:26:01 EDT 2026 [ 350.682811] Lustre: DEBUG MARKER: excepting tests: 18b 23b [ 362.338470] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 372.206837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 372.217670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.250335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 374.240936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 374.241424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.251714] Lustre: Skipped 1 previous similar message [ 374.295527] Lustre: Skipped 1 previous similar message [ 376.622695] Lustre: server umount lustre-MDT0000 complete [ 384.488488] LustreError: 8459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 384.526325] LustreError: 8459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 386.036132] LustreError: 10338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838021 with bad export cookie 9785422548748180143 [ 386.039225] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 386.060256] LustreError: 10338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 386.526716] Lustre: server umount lustre-MDT0001 complete [ 405.387422] Lustre: server umount lustre-OST0000 complete [ 405.985696] Lustre: 3645:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838025/real 1788838025] req@ffff9d16f0dd3100 x1875732341046272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788838041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 406.030757] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 406.058230] Lustre: Skipped 1 previous similar message [ 408.037325] Lustre: 3646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838027/real 1788838027] req@ffff9d16edfb9880 x1875732341046528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788838043 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 411.105329] Lustre: 3644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838030/real 1788838030] req@ffff9d16f0dd2300 x1875732341046784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788838046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 415.338993] Lustre: server umount lustre-OST0001 complete [ 431.504738] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 434.503269] Key type lgssc unregistered [ 434.783640] LNet: 14799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.796687] LNetError: 14799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 435.818932] LNet: Removed LNI 192.168.203.110@tcp [ 436.961206] Key type .llcrypt unregistered [ 436.965496] Key type ._llcrypt unregistered [ 465.451390] Key type ._llcrypt registered [ 465.456936] Key type .llcrypt registered [ 465.635369] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 483.473441] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 484.735195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 484.812909] alg: No test for adler32 (adler32-zlib) [ 485.906443] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 486.189866] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 487.896302] Key type lgssc registered [ 489.574680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 540.069477] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 552.216891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 552.244325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.565581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 553.598282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 553.681907] Lustre: lustre-MDT0000: new disk, initializing [ 553.749606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.767114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 557.704515] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 569.479908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.612089] Lustre: 19248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 569.658749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 569.663288] Lustre: Skipped 1 previous similar message [ 569.734406] Lustre: lustre-MDT0001: new disk, initializing [ 569.803659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 569.834767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 569.839700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 573.753185] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 578.330613] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 587.634916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 587.897136] Lustre: lustre-OST0000: new disk, initializing [ 587.900772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 587.905965] Lustre: 21186:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 587.987549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 593.450425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 593.455329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 593.528867] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 594.198976] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 609.403577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 609.634541] Lustre: lustre-OST0001: new disk, initializing [ 609.640677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 609.653539] Lustre: 22209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 609.743275] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 617.116978] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 619.571950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 619.588487] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 619.629684] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 629.332701] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 636.369800] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 644.908646] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 23:31:18 (1788838278) === [ 647.803333] Lustre: DEBUG MARKER: == sanity-lfsck test 0: Control LFSCK manually =========== 23:31:21 (1788838281) [ 647.941156] Lustre: 19256:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 647.948400] Lustre: 19256:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 647.955543] Lustre: 19256:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 647.964702] Lustre: 19256:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 647.972429] Lustre: 19256:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 647.977469] Lustre: 19256:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 648.475779] Lustre: 19255:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 648.481647] Lustre: 19255:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 14 previous similar messages [ 648.485913] Lustre: 19255:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 648.490540] Lustre: 19255:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 14 previous similar messages [ 648.494724] Lustre: 19255:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 648.500193] Lustre: 19255:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 14 previous similar messages [ 648.505280] Lustre: 19255:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 648.515753] Lustre: 19255:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 14 previous similar messages [ 648.522191] Lustre: 19255:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 648.526740] Lustre: 19255:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 14 previous similar messages [ 648.532958] Lustre: 19255:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 648.540497] Lustre: 19255:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 14 previous similar messages [ 649.545680] Lustre: 22836:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 649.559795] Lustre: 22836:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 50 previous similar messages [ 649.570948] Lustre: 22836:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 649.581254] Lustre: 22836:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 50 previous similar messages [ 649.588930] Lustre: 22836:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 649.606276] Lustre: 22836:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 50 previous similar messages [ 649.613204] Lustre: 22836:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 649.622505] Lustre: 22836:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 50 previous similar messages [ 649.636698] Lustre: 22836:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 649.652258] Lustre: 22836:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 50 previous similar messages [ 649.662221] Lustre: 22836:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 649.671201] Lustre: 22836:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 50 previous similar messages [ 651.561433] Lustre: 22836:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 651.574778] Lustre: 22836:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 89 previous similar messages [ 651.581993] Lustre: 22836:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 651.591556] Lustre: 22836:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 89 previous similar messages [ 651.610284] Lustre: 22836:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 651.616458] Lustre: 22836:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 89 previous similar messages [ 651.629565] Lustre: 22836:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 651.639156] Lustre: 22836:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 89 previous similar messages [ 651.652548] Lustre: 22836:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 651.662854] Lustre: 22836:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 89 previous similar messages [ 651.673490] Lustre: 22836:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 651.681807] Lustre: 22836:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 89 previous similar messages [ 656.214537] Lustre: *** cfs_fail_loc=1600, val=3*** [ 658.436238] Lustre: 21175:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 274, rollback = 2 [ 658.449972] Lustre: 21175:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 121 previous similar messages [ 658.462174] Lustre: 21175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 658.474144] Lustre: 23604:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 658.476068] Lustre: 21175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 122 previous similar messages [ 658.476096] Lustre: 21175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 658.476100] Lustre: 21175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 121 previous similar messages [ 658.476105] Lustre: 21175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 658.495702] Lustre: 23604:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 122 previous similar messages [ 658.495743] Lustre: 23604:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 658.495747] Lustre: 23604:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 121 previous similar messages [ 658.600208] Lustre: 21175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 127 previous similar messages [ 661.045472] Lustre: *** cfs_fail_loc=1600, val=3*** [ 674.316181] Lustre: 23604:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 2: before 259 < left 274, rollback = 2 [ 674.321917] Lustre: 23400:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 674.344768] Lustre: 23604:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 62 previous similar messages [ 674.344796] Lustre: 23604:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 674.344800] Lustre: 23604:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 60 previous similar messages [ 674.344806] Lustre: 23604:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 674.344809] Lustre: 23604:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 61 previous similar messages [ 674.344815] Lustre: 23604:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 674.344818] Lustre: 23604:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 55 previous similar messages [ 674.344822] Lustre: 23604:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 674.344825] Lustre: 23604:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 61 previous similar messages [ 674.501831] Lustre: 23400:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 63 previous similar messages [ 677.355494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 677.375295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.395048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 680.929809] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 680.930254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 680.957250] Lustre: Skipped 2 previous similar messages [ 680.969355] Lustre: Skipped 3 previous similar messages [ 683.935183] Lustre: server umount lustre-MDT0000 complete [ 688.160337] LustreError: 19241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838324 with bad export cookie 9540182920477207679 [ 688.161441] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 688.171583] LustreError: 19241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 688.596664] Lustre: server umount lustre-MDT0001 complete [ 703.496566] Lustre: server umount lustre-OST0000 complete [ 717.765872] Lustre: server umount lustre-OST0001 complete [ 726.556173] Lustre: DEBUG MARKER: == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 23:32:39 (1788838359) [ 742.061067] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 753.579315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 754.163686] LustreError: 26273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 754.191585] LustreError: 26273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 754.279491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 759.110612] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 759.785886] LustreError: 26274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 764.898376] LustreError: 26273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 769.766720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.017360] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 770.234197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 775.717142] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 779.282542] Lustre: 27413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 786.488081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 786.843647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.641160] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 796.069339] LustreError: 27769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 799.166551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 800.923252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 806.395208] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:40 to 0x2c0000401:65) [ 807.713960] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 814.946531] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 818.776846] Lustre: 29280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 820.308231] Lustre: 27477:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 820.317346] Lustre: 27477:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 820.322397] Lustre: 27477:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 820.331661] Lustre: 27477:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 820.346250] Lustre: 27477:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 820.358709] Lustre: 27477:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 820.368740] Lustre: 27477:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 820.377725] Lustre: 27477:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 820.386954] Lustre: 27477:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 820.399374] Lustre: 27477:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 820.406885] Lustre: 27477:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 820.413995] Lustre: 27477:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 825.987040] Lustre: *** cfs_fail_loc=1501, val=0*** [ 834.834442] Lustre: Failing over lustre-MDT0000 [ 835.167849] Lustre: server umount lustre-MDT0000 complete [ 836.576861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 836.581904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.590541] LustreError: 26273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 836.618034] LustreError: 26273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 845.871130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 845.965714] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 846.177994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.185803] Lustre: Skipped 1 previous similar message [ 846.214812] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 851.048178] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 851.427686] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 851.431568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 851.449762] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 851.487326] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 851.489271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 854.333367] Lustre: *** cfs_fail_loc=1505, val=0*** [ 863.088558] Lustre: DEBUG MARKER: == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 23:34:56 (1788838496) [ 864.297578] Lustre: 27477:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 864.304987] Lustre: 27477:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 322 previous similar messages [ 864.309447] Lustre: 27477:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 864.314816] Lustre: 27477:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 864.320963] Lustre: 27477:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 864.327129] Lustre: 27477:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 864.331656] Lustre: 27477:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 864.335591] Lustre: 27477:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 864.340382] Lustre: 27477:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 864.344431] Lustre: 27477:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 864.349330] Lustre: 27477:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 864.353555] Lustre: 27477:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 870.164336] Lustre: *** cfs_fail_loc=1502, val=0*** [ 882.275186] Lustre: Failing over lustre-MDT0000 [ 882.659913] Lustre: server umount lustre-MDT0000 complete [ 887.264812] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.267502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 887.285606] Lustre: Skipped 3 previous similar messages [ 887.292272] LustreError: 26269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.339213] LustreError: 26269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 894.466496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 894.715814] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 895.059885] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 900.077198] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 900.081057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 900.101636] Lustre: Skipped 3 previous similar messages [ 900.127086] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 900.158775] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:169 to 0x2c0000401:193) [ 900.160287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 900.963575] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 905.317081] Lustre: *** cfs_fail_loc=1505, val=0*** [ 913.661755] Lustre: DEBUG MARKER: == sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 23:35:47 (1788838547) [ 921.027148] Lustre: *** cfs_fail_loc=1504, val=0*** [ 921.035801] Lustre: *** cfs_fail_loc=1504, val=0*** [ 921.044840] Lustre: Skipped 1 previous similar message [ 929.041982] Lustre: Failing over lustre-MDT0000 [ 929.259043] Lustre: server umount lustre-MDT0000 complete [ 930.785739] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.789799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 930.794576] LustreError: 27477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.794588] LustreError: 27477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 930.799728] Lustre: Skipped 6 previous similar messages [ 939.339670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 939.453660] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 939.642483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 939.650964] Lustre: Skipped 1 previous similar message [ 939.683334] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 943.916862] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 945.131274] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 945.134784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 945.162291] Lustre: Skipped 3 previous similar messages [ 945.190612] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 945.233973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:233 to 0x280000401:257) [ 945.234192] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:257) [ 946.936172] Lustre: *** cfs_fail_loc=1505, val=0*** [ 953.956416] Lustre: DEBUG MARKER: == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 23:36:27 (1788838587) [ 955.326814] Lustre: 26269:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 955.334215] Lustre: 26269:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 644 previous similar messages [ 955.343701] Lustre: 26269:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 955.349871] Lustre: 26269:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 955.355570] Lustre: 26269:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 955.365108] Lustre: 26269:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 955.380414] Lustre: 26269:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 955.386720] Lustre: 26269:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 955.396382] Lustre: 26269:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 955.401749] Lustre: 26269:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 955.417285] Lustre: 26269:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 955.435622] Lustre: 26269:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 961.017618] Lustre: *** cfs_fail_loc=1603, val=0*** [ 969.535028] Lustre: Failing over lustre-MDT0000 [ 970.370536] Lustre: server umount lustre-MDT0000 complete [ 970.724924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.740862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 970.748753] LustreError: 30893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.748764] LustreError: 30893:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 982.862892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 983.017126] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 983.280641] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 988.499918] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 988.645461] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 988.659349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 988.664354] Lustre: Skipped 3 previous similar messages [ 988.683989] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 988.750170] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:297 to 0x2c0000401:321) [ 988.751182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:297 to 0x280000401:321) [ 998.515199] Lustre: DEBUG MARKER: == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 23:37:12 (1788838632) [ 1004.869324] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1011.787411] Lustre: Failing over lustre-MDT0000 [ 1012.124598] Lustre: server umount lustre-MDT0000 complete [ 1012.196179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.211115] Lustre: Skipped 5 previous similar messages [ 1022.768856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1022.864380] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.274868] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1027.507706] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1028.579229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1028.580757] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1028.590694] Lustre: Skipped 3 previous similar messages [ 1028.619718] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1028.657199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:361 to 0x2c0000401:385) [ 1028.660214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:361 to 0x280000401:385) [ 1037.437495] Lustre: DEBUG MARKER: == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 23:37:50 (1788838670) [ 1044.645944] Lustre: *** cfs_fail_loc=1605, val=0*** [ 1052.628421] Lustre: Failing over lustre-MDT0000 [ 1052.870602] Lustre: server umount lustre-MDT0000 complete [ 1054.184226] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1054.188304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1054.188680] LustreError: 26269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1054.188689] LustreError: 26269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 1054.198750] Lustre: Skipped 3 previous similar messages [ 1063.097545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1063.248359] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1063.634408] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1068.711559] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1069.026438] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1069.032661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1069.059494] Lustre: Skipped 3 previous similar messages [ 1069.074632] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1069.120926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:425 to 0x280000401:449) [ 1069.122452] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:425 to 0x2c0000401:449) [ 1079.817410] Lustre: DEBUG MARKER: == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 23:38:32 (1788838712) [ 1083.355984] Lustre: 26268:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 1083.364573] Lustre: 26268:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1070 previous similar messages [ 1083.381163] Lustre: 26268:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1083.389481] Lustre: 26268:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1070 previous similar messages [ 1083.398578] Lustre: 26268:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1083.406596] Lustre: 26268:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1070 previous similar messages [ 1083.416696] Lustre: 26268:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 1083.422726] Lustre: 26268:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1070 previous similar messages [ 1083.427376] Lustre: 26268:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1083.430800] Lustre: 26268:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1070 previous similar messages [ 1083.435926] Lustre: 26268:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1083.441420] Lustre: 26268:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1070 previous similar messages [ 1087.112680] Lustre: *** cfs_fail_loc=161d, val=0*** [ 1096.381895] Lustre: Failing over lustre-MDT0000 [ 1096.627777] Lustre: server umount lustre-MDT0000 complete [ 1106.367886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1106.526690] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1106.723402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1106.726854] Lustre: Skipped 3 previous similar messages [ 1106.768785] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1111.652682] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1112.043373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1112.043602] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1112.051020] Lustre: Skipped 3 previous similar messages [ 1112.066488] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1112.102513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:489 to 0x280000401:513) [ 1112.110165] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 1120.199336] Lustre: DEBUG MARKER: == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 23:39:13 (1788838753) [ 1123.033490] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1132.509838] Lustre: DEBUG MARKER: == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 23:39:26 (1788838766) [ 1138.776049] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1139.618561] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1149.864915] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 23:39:43 (1788838783) [ 1184.957613] Lustre: Failing over lustre-MDT0000 [ 1185.300480] Lustre: server umount lustre-MDT0000 complete [ 1188.842141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.866931] Lustre: Skipped 5 previous similar messages [ 1188.882838] LustreError: 26269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.906262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1188.945786] LustreError: 26269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 1191.670891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1201.726592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1204.698141] Lustre: 16408:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838824/real 1788838824] req@ffff9d16ed54ce00 x1875732723363712/t0(0) o400->MGC192.168.203.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788838840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1204.724393] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1212.636112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1212.670702] Lustre: lustre-MDT0000: reset Object Index mappings [ 1214.880178] LustreError: 42438:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1214.893457] LustreError: 42438:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9d16fd04ea00 x1875732723369728/t0(0) o250->MGC192.168.203.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788838850 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1214.922507] LustreError: 42438:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1214.945879] LustreError: 16407:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d16ed54ce00 x1875732723372416/t0(0) o250->MGC192.168.203.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1215.292802] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1219.283346] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1220.584467] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1220.617226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1220.632963] Lustre: Skipped 3 previous similar messages [ 1220.654229] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1220.694790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:609) [ 1220.695420] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:609) [ 1223.463910] LustreError: 42924:0:(lfsck_engine.c:1018:lfsck_master_engine()) lustre-MDT0000-osd: master engine fail to verify the .lustre/lost+found/, go ahead: rc = -115 [ 1223.483117] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1225.570433] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1225.579867] Lustre: Skipped 1 previous similar message [ 1232.120959] Lustre: Failing over lustre-MDT0000 [ 1232.327080] Lustre: server umount lustre-MDT0000 complete [ 1235.939071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1242.976886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1247.705726] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1248.284138] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:641) [ 1248.285242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:641) [ 1250.999556] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1258.289649] Lustre: DEBUG MARKER: == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 23:41:31 (1788838891) [ 1260.842358] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1271.160140] Lustre: Failing over lustre-MDT0000 [ 1271.441381] Lustre: server umount lustre-MDT0000 complete [ 1277.096084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1289.167431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1290.211910] Lustre: 16408:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838909/real 1788838909] req@ffff9d16f4f10000 x1875732723460480/t0(0) o400->MGC192.168.203.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788838925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1303.166695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1303.196963] Lustre: lustre-MDT0000: reset Object Index mappings [ 1315.817434] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x846589d1f34d47c8 [ 1315.841935] Lustre: MGC192.168.203.110@tcp: Connection restored to 0@lo (at 0@lo) [ 1315.865303] Lustre: Skipped 7 previous similar messages [ 1316.301662] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1316.310926] Lustre: Skipped 1 previous similar message [ 1321.443860] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1321.452324] Lustre: Skipped 1 previous similar message [ 1321.473246] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1321.483318] Lustre: Skipped 1 previous similar message [ 1321.539831] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:705) [ 1321.542979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:705) [ 1322.018845] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1326.534033] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1326.535602] Lustre: Skipped 2 previous similar messages [ 1338.755994] Lustre: Failing over lustre-MDT0000 [ 1339.129159] Lustre: server umount lustre-MDT0000 complete [ 1341.921756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1341.932584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.960724] Lustre: Skipped 13 previous similar messages [ 1349.883979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1350.063545] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1350.090473] LustreError: Skipped 2 previous similar messages [ 1354.708864] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1355.846322] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:737) [ 1355.847518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:737) [ 1358.192579] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1358.198433] Lustre: Skipped 84 previous similar messages [ 1365.184710] Lustre: DEBUG MARKER: == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 23:43:18 (1788838998) [ 1366.641987] Lustre: 27477:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 1366.647194] Lustre: 27477:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1181 previous similar messages [ 1366.651617] Lustre: 27477:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 1366.656565] Lustre: 27477:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1181 previous similar messages [ 1366.661740] Lustre: 27477:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1366.668223] Lustre: 27477:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1181 previous similar messages [ 1366.674507] Lustre: 27477:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1366.686271] Lustre: 27477:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1181 previous similar messages [ 1366.693946] Lustre: 27477:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1366.704592] Lustre: 27477:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1181 previous similar messages [ 1366.714873] Lustre: 27477:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1366.721676] Lustre: 27477:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1181 previous similar messages [ 1374.036942] Lustre: *** cfs_fail_loc=1600, val=1*** [ 1374.042167] Lustre: Skipped 7 previous similar messages [ 1396.879793] Lustre: DEBUG MARKER: == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 23:43:50 (1788839030) [ 1404.692136] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1404.695740] Lustre: Skipped 10 previous similar messages [ 1429.204957] Lustre: DEBUG MARKER: == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 23:44:22 (1788839062) [ 1441.037153] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1441.043411] Lustre: Skipped 12 previous similar messages [ 1447.215259] Lustre: Failing over lustre-MDT0000 [ 1447.794361] Lustre: server umount lustre-MDT0000 complete [ 1447.911659] LustreError: 29294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1447.936203] LustreError: 29294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 80 previous similar messages [ 1460.122646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1460.721342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.749553] Lustre: Skipped 4 previous similar messages [ 1460.833226] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1460.855105] Lustre: Skipped 1 previous similar message [ 1465.832303] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1465.842684] Lustre: Skipped 1 previous similar message [ 1465.857157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1465.863119] Lustre: Skipped 8 previous similar messages [ 1465.893134] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1465.899145] Lustre: Skipped 1 previous similar message [ 1465.940855] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:854 to 0x2c0000401:897) [ 1465.949391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:853 to 0x280000401:897) [ 1466.519786] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1476.781404] Lustre: DEBUG MARKER: == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 23:45:10 (1788839110) [ 1491.615298] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 1509.594873] Lustre: 52893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1533.354401] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1536.747267] Lustre: 54029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1544.259926] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1544.278817] Lustre: Skipped 81 previous similar messages [ 1547.551934] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1548.577410] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1549.600268] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1551.649146] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1551.652385] Lustre: Skipped 1 previous similar message [ 1552.523514] Lustre: Failing over lustre-MDT0000 [ 1552.867165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1552.909861] Lustre: server umount lustre-MDT0000 complete [ 1561.743754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1567.314719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:946 to 0x280000401:961) [ 1567.315303] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:947 to 0x2c0000401:993) [ 1567.653101] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1579.187350] Lustre: DEBUG MARKER: == sanity-lfsck test 8: LFSCK state machine ============== 23:46:52 (1788839212) [ 1582.566920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1582.583572] Lustre: Skipped 3 previous similar messages [ 1587.681783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1587.695101] Lustre: Skipped 3 previous similar messages [ 1587.871464] Lustre: server umount lustre-MDT0000 complete [ 1591.910036] LustreError: 26254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839227 with bad export cookie 9540182920477422278 [ 1591.939877] LustreError: 26254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1592.397640] Lustre: server umount lustre-MDT0001 complete [ 1607.134865] Lustre: server umount lustre-OST0000 complete [ 1608.160646] Lustre: 16409:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839228/real 1788839228] req@ffff9d15c2e8d880 x1875732723796608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788839244 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1608.192101] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.213609] Lustre: Skipped 12 previous similar messages [ 1611.285798] Lustre: server umount lustre-OST0001 complete [ 1618.989565] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 1625.909389] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 1668.862629] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 1679.190205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.433667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1679.461850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1679.559609] Lustre: lustre-MDT0000: new disk, initializing [ 1679.702342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1684.386211] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1694.818643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.927619] Lustre: 59092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1694.957655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1694.962140] Lustre: Skipped 1 previous similar message [ 1695.027617] Lustre: lustre-MDT0001: new disk, initializing [ 1695.140728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1695.157876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1699.528985] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1704.165305] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1711.742726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1711.990358] Lustre: lustre-OST0000: new disk, initializing [ 1711.996246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1712.003431] Lustre: 60726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1713.083202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1713.109930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1713.180828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1719.609688] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1733.098120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1733.179979] Lustre: lustre-OST0001: new disk, initializing [ 1733.182834] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1733.187282] Lustre: 61597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1734.822990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1734.835887] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1734.910564] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1740.717702] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1750.850514] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1754.528770] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1764.690020] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1765.575643] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1765.579201] Lustre: Skipped 19 previous similar messages [ 1769.438597] Lustre: *** cfs_fail_loc=1601, val=2*** [ 1769.448417] Lustre: Skipped 6 previous similar messages [ 1786.433759] Lustre: Failing over lustre-MDT0000 [ 1786.864471] Lustre: server umount lustre-MDT0000 complete [ 1798.969965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1799.163945] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1799.172849] LustreError: Skipped 3 previous similar messages [ 1799.663567] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1799.684052] Lustre: Skipped 1 previous similar message [ 1804.780148] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1804.801306] Lustre: Skipped 1 previous similar message [ 1804.805050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1804.825065] Lustre: Skipped 7 previous similar messages [ 1804.843587] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1804.857047] Lustre: Skipped 1 previous similar message [ 1804.914878] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1804.921729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:65) [ 1804.925340] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:65) [ 1805.483074] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1814.023463] Lustre: Failing over lustre-MDT0000 [ 1815.010415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1815.033200] LustreError: Skipped 2 previous similar messages [ 1816.243638] Lustre: server umount lustre-MDT0000 complete [ 1825.352141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1830.103183] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1830.932369] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1830.949569] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:97) [ 1830.950622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:97) [ 1835.227959] Lustre: Failing over lustre-MDT0000 [ 1835.447257] Lustre: server umount lustre-MDT0000 complete [ 1844.566688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1850.120807] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1850.415500] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:129) [ 1850.416762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:129) [ 1857.141175] Lustre: *** cfs_fail_loc=1602, val=2*** [ 1875.455566] Lustre: DEBUG MARKER: == sanity-lfsck test 9a: LFSCK speed control (1) ========= 23:51:48 (1788839508) [ 1890.660549] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 1908.048145] Lustre: 68556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1932.669372] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1936.517830] Lustre: 69694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1945.767515] Lustre: 61025:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1945.774044] Lustre: 61025:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 2835 previous similar messages [ 1945.785553] Lustre: 61025:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1945.792747] Lustre: 61025:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 2836 previous similar messages [ 1945.798668] Lustre: 61025:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1945.804432] Lustre: 61025:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 2836 previous similar messages [ 1945.813408] Lustre: 61025:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1945.818760] Lustre: 61025:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 2836 previous similar messages [ 1945.828138] Lustre: 61025:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1945.833705] Lustre: 61025:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 2836 previous similar messages [ 1945.839472] Lustre: 61025:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1945.844698] Lustre: 61025:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 2836 previous similar messages [ 2058.287205] Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) ========= 23:54:51 (1788839691) [ 2109.347341] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2109.352175] Lustre: Skipped 4 previous similar messages [ 2134.603320] Lustre: *** cfs_fail_loc=160c, val=0*** [ 2134.611389] Lustre: Skipped 8 previous similar messages [ 2172.934646] Lustre: DEBUG MARKER: == sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 23:56:46 (1788839806) [ 2223.338274] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2224.365045] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2224.373908] Lustre: Skipped 36 previous similar messages [ 2226.373681] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2226.375799] Lustre: Skipped 76 previous similar messages [ 2230.376940] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2230.381447] Lustre: Skipped 160 previous similar messages [ 2238.377550] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2238.382559] Lustre: Skipped 315 previous similar messages [ 2254.395519] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2254.399388] Lustre: Skipped 782 previous similar messages [ 2286.425760] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2286.436311] Lustre: Skipped 1580 previous similar messages [ 2300.357336] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2300.364538] Lustre: Skipped 2599 previous similar messages [ 2550.280182] Lustre: DEBUG MARKER: == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 00:03:03 (1788840183) [ 2600.196383] Lustre: 59103:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 320, rollback = 2 [ 2600.204624] Lustre: 59103:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 36831 previous similar messages [ 2600.210606] Lustre: 59103:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 1/4/0 [ 2600.219103] Lustre: 59103:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2600.226662] Lustre: 59103:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 5/5/0, xattr_set: 7/320/0 [ 2600.231851] Lustre: 59103:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2600.240547] Lustre: 59103:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 6/46/0, punch: 0/0/0, quota 1/3/0 [ 2600.247551] Lustre: 59103:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2600.252632] Lustre: 59103:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 2/5/0 [ 2600.259051] Lustre: 59103:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2600.265343] Lustre: 59103:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1 [ 2600.270666] Lustre: 59103:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2720.765786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.766180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2720.778901] Lustre: Skipped 13 previous similar messages [ 2720.792755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2720.800861] LustreError: Skipped 1 previous similar message [ 2722.878534] Lustre: server umount lustre-MDT0000 complete [ 2725.857840] LustreError: 59103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.872563] LustreError: 59103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 48 previous similar messages [ 2726.614410] LustreError: 60725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840362 with bad export cookie 9540182920477441381 [ 2726.615733] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2726.630064] LustreError: 60725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2726.657359] LustreError: Skipped 2 previous similar messages [ 2726.910353] Lustre: server umount lustre-MDT0001 complete [ 2740.376575] Lustre: server umount lustre-OST0000 complete [ 2754.176708] Lustre: server umount lustre-OST0001 complete [ 2760.558041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 2768.954557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2784.544590] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2789.664244] LustreError: 74846:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.110@tcp: failed processing log, type 4: rc = -110 [ 2815.392181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.413674] Lustre: Skipped 8 previous similar messages [ 2822.445950] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2826.105669] Lustre: 75428:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2826.123817] Lustre: *** cfs_fail_loc=160e, val=3*** [ 2829.166729] Lustre: 75428:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2838.814745] Lustre: DEBUG MARKER: == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 00:07:52 (1788840472) [ 2853.091674] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 2863.231046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.690957] LustreError: 74871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.832627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3272 to 0x280000401:3297) [ 2867.294420] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2874.793405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.333084] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2882.125341] Lustre: 78095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2895.645994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2895.858446] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2895.868583] Lustre: Skipped 2 previous similar messages [ 2900.970475] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3207 to 0x2c0000401:3233) [ 2901.285694] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2908.383723] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2911.621310] Lustre: 79590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2915.373338] Lustre: *** cfs_fail_loc=160d, val=0*** [ 2921.427676] Lustre: Failing over lustre-OST0000 [ 2921.441474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2921.450586] Lustre: Skipped 4 previous similar messages [ 2921.554203] Lustre: server umount lustre-OST0000 complete [ 2929.139955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2929.423121] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2929.434847] Lustre: Skipped 2 previous similar messages [ 2931.176696] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2931.184109] Lustre: Skipped 2 previous similar messages [ 2931.203592] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2931.206692] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2931.208515] Lustre: *** cfs_fail_loc=215, val=0*** [ 2931.210942] Lustre: Skipped 2 previous similar messages [ 2931.219071] Lustre: Skipped 11 previous similar messages [ 2935.636853] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2936.288794] Lustre: *** cfs_fail_loc=215, val=0*** [ 2936.293146] Lustre: Skipped 2 previous similar messages [ 2939.420762] Lustre: 80991:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2939.446846] Lustre: 80991:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2941.408380] Lustre: *** cfs_fail_loc=215, val=0*** [ 2941.410552] Lustre: Skipped 1 previous similar message [ 2941.992959] Lustre: Failing over lustre-OST0000 [ 2942.230901] Lustre: server umount lustre-OST0000 complete [ 2951.301580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2952.959574] Lustre: *** cfs_fail_loc=215, val=0*** [ 2958.255498] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2958.304442] Lustre: *** cfs_fail_loc=215, val=0*** [ 2958.311055] Lustre: Skipped 2 previous similar messages [ 2967.011657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2967.029207] Lustre: Skipped 3 previous similar messages [ 2972.138879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.148141] Lustre: Skipped 3 previous similar messages [ 2977.249054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2977.253549] Lustre: Skipped 3 previous similar messages [ 2979.296134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2979.529233] Lustre: server umount lustre-MDT0000 complete [ 2983.068777] LustreError: 74852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840618 with bad export cookie 9540182920479018831 [ 2983.081921] LustreError: 74852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2983.300457] Lustre: server umount lustre-MDT0001 complete [ 2996.856634] Lustre: server umount lustre-OST0000 complete [ 3009.345752] Lustre: server umount lustre-OST0001 complete [ 3017.338475] Lustre: DEBUG MARKER: == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 00:10:51 (1788840651) [ 3030.274546] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 3039.097119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.541460] LustreError: 84235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.566286] LustreError: 84235:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 3039.666588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.682359] Lustre: Skipped 2 previous similar messages [ 3043.735215] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3052.359400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.697596] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3059.479090] Lustre: 85378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3065.399400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3071.710891] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3079.829714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3085.103320] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3207 to 0x2c0000401:3265) [ 3085.136506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3362 to 0x280000401:3393) [ 3086.108980] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3093.874157] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3097.351649] Lustre: 87249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3159.751944] Lustre: DEBUG MARKER: == sanity-lfsck test 12b: auto detect Lustre device ====== 00:13:13 (1788840793) [ 3174.421237] Lustre: DEBUG MARKER: == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 00:13:27 (1788840807) [ 3175.628478] Lustre: *** cfs_fail_loc=160f, val=0*** [ 3186.253390] Lustre: DEBUG MARKER: == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 00:13:39 (1788840819) [ 3190.109953] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3190.117430] Lustre: Skipped 7 previous similar messages [ 3205.289507] Lustre: 84235:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 252 < left 260, rollback = 2 [ 3205.301448] Lustre: 84235:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1324 previous similar messages [ 3205.309031] Lustre: 84235:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3205.317562] Lustre: 84235:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1324 previous similar messages [ 3205.324725] Lustre: 84235:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/260/0 [ 3205.330611] Lustre: 84235:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1324 previous similar messages [ 3205.337042] Lustre: 84235:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 0/0/0, quota 1/3/2 [ 3205.342972] Lustre: 84235:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1324 previous similar messages [ 3205.348318] Lustre: 84235:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 1/17/2, delete: 0/0/0 [ 3205.353938] Lustre: 84235:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1324 previous similar messages [ 3205.363670] Lustre: 84235:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3205.370337] Lustre: 84235:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1324 previous similar messages [ 3241.954154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3246.233731] Lustre: server umount lustre-MDT0000 complete [ 3250.906706] LustreError: 84216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840886 with bad export cookie 9540182920479027329 [ 3250.918915] LustreError: 84216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3251.333807] Lustre: server umount lustre-MDT0001 complete [ 3265.878524] Lustre: server umount lustre-OST0000 complete [ 3280.001888] Lustre: server umount lustre-OST0001 complete [ 3296.336486] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 3306.974736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.367938] LustreError: 92784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3307.388722] LustreError: 92784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3307.450492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.460935] Lustre: Skipped 3 previous similar messages [ 3312.094292] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3320.875335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.633873] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3328.579157] Lustre: 93926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3335.108897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3342.923323] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3346.671318] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:161) [ 3352.047388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3353.192467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3554 to 0x280000401:3585) [ 3353.221298] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3361) [ 3353.313409] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:161) [ 3359.668535] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3368.380080] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3372.485325] Lustre: 95798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3380.363602] Lustre: DEBUG MARKER: == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 00:16:53 (1788841013) [ 3386.930419] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3386.944275] Lustre: Skipped 63 previous similar messages [ 3413.475164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3413.488379] LustreError: Skipped 2 previous similar messages [ 3413.499943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.516024] Lustre: Skipped 14 previous similar messages [ 3413.529822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.539240] Lustre: Skipped 4 previous similar messages [ 3416.913375] Lustre: server umount lustre-MDT0000 complete [ 3420.659158] LustreError: 92765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841056 with bad export cookie 9540182920479055728 [ 3420.666319] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3420.679543] LustreError: 92765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3420.701301] LustreError: Skipped 2 previous similar messages [ 3421.010870] Lustre: server umount lustre-MDT0001 complete [ 3434.625208] Lustre: server umount lustre-OST0000 complete [ 3448.924362] Lustre: server umount lustre-OST0001 complete [ 3466.262904] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 3477.553055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.346784] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3493.535053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.799279] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3500.928457] Lustre: 99833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3507.259773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3513.986198] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3519.724267] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:193) [ 3522.806089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3682 to 0x280000401:3713) [ 3523.018397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3528.700906] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:193) [ 3528.709448] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3393) [ 3529.838620] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3537.723699] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3547.958963] Lustre: DEBUG MARKER: == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 00:19:41 (1788841181) [ 3552.046228] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3552.048369] Lustre: Skipped 63 previous similar messages [ 3552.358400] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3564.204254] Lustre: DEBUG MARKER: == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 00:19:57 (1788841197) [ 3566.938261] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3567.018972] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3567.025535] Lustre: Skipped 2 previous similar messages [ 3580.682265] Lustre: DEBUG MARKER: == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 00:20:13 (1788841213) [ 3582.195778] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_15c MDS newer than 2.7.55, LU-6475 [ 3584.194779] Lustre: DEBUG MARKER: == sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 00:20:17 (1788841217) [ 3590.554447] Lustre: *** cfs_fail_loc=1709, val=0*** [ 3590.558890] LustreError: 102908:0:(mdt_reint.c:2644:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240002340:0x6c:0x0]/f60 failed: rc = -5 [ 3661.794188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.802498] Lustre: Skipped 6 previous similar messages [ 3667.920096] Lustre: server umount lustre-MDT0000 complete [ 3679.130568] LustreError: 98673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841315 with bad export cookie 9540182920479070456 [ 3679.157333] LustreError: 98673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3679.408078] Lustre: server umount lustre-MDT0001 complete [ 3698.464242] Lustre: 16410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841318/real 1788841318] req@ffff9d15cf539180 x1875732728728576/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841334 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3700.625209] Lustre: server umount lustre-OST0000 complete [ 3700.643094] Lustre: 16408:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841320/real 1788841320] req@ffff9d16f8af8e00 x1875732728728832/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841336 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3703.649159] Lustre: 16410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841323/real 1788841323] req@ffff9d15cf53ad80 x1875732728729088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841339 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3705.824765] Lustre: 16408:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841325/real 1788841325] req@ffff9d15cf538e00 x1875732728729472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3709.477245] Lustre: server umount lustre-OST0001 complete [ 3729.772292] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 3732.584331] Key type lgssc unregistered [ 3732.991724] LNet: 105542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3733.017956] LNetError: 105542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3733.028961] LNet: Removed LNI 192.168.203.110@tcp [ 3734.143153] Key type .llcrypt unregistered [ 3734.147154] Key type ._llcrypt unregistered [ 3763.081190] Key type ._llcrypt registered [ 3763.090892] Key type .llcrypt registered [ 3763.184689] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 3777.258517] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 3778.286635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3778.411478] alg: No test for adler32 (adler32-zlib) [ 3779.531656] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 3779.948721] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 3781.696205] Key type lgssc registered [ 3782.879657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3836.788876] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 3849.713956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3849.751728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3851.044833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3851.097841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3851.198068] Lustre: lustre-MDT0000: new disk, initializing [ 3851.301413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3851.340819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3856.448601] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3870.896937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.040975] Lustre: 109994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3871.106854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3871.115420] Lustre: Skipped 1 previous similar message [ 3871.212801] Lustre: lustre-MDT0001: new disk, initializing [ 3871.274460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3871.312932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3871.344708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3876.024712] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3881.039353] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3892.911154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3893.260374] Lustre: lustre-OST0000: new disk, initializing [ 3893.265384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3893.272429] Lustre: 111930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3893.353433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3899.719425] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3902.492686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3902.505369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3902.624482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3913.485615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3913.626817] Lustre: lustre-OST0001: new disk, initializing [ 3913.631152] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3913.642567] Lustre: 112956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3913.690074] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3920.877273] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3921.998061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3922.021866] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3922.106829] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3932.598894] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3940.999892] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3947.129871] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 00:26:20 (1788841580) === [ 3953.485784] Lustre: DEBUG MARKER: == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 00:26:27 (1788841587) [ 3953.746482] Lustre: 110001:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 3953.765944] Lustre: 110001:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3953.772138] Lustre: 110001:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3953.780117] Lustre: 110001:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 3953.789086] Lustre: 110001:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 3953.798665] Lustre: 110001:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3954.330592] Lustre: 110000:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 3954.337703] Lustre: 110000:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 9 previous similar messages [ 3954.345610] Lustre: 110000:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3954.350690] Lustre: 110000:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3954.357934] Lustre: 110000:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3954.363860] Lustre: 110000:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3954.370628] Lustre: 110000:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 7/369/2 [ 3954.376782] Lustre: 110000:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3954.382677] Lustre: 110000:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 3954.388249] Lustre: 110000:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3954.395620] Lustre: 110000:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3954.401359] Lustre: 110000:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3955.335629] Lustre: 110001:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 3955.343763] Lustre: 110001:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 206 previous similar messages [ 3955.350122] Lustre: 110001:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 3955.359355] Lustre: 110001:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 206 previous similar messages [ 3955.365353] Lustre: 110001:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3955.371924] Lustre: 110001:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 206 previous similar messages [ 3955.378937] Lustre: 110001:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 7/369/0 [ 3955.384889] Lustre: 110001:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 206 previous similar messages [ 3955.391385] Lustre: 110001:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 3955.402681] Lustre: 110001:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 206 previous similar messages [ 3955.410340] Lustre: 110001:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3955.416722] Lustre: 110001:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 206 previous similar messages [ 3956.888543] Lustre: *** cfs_fail_loc=1613, val=0*** [ 3968.477755] Lustre: DEBUG MARKER: == sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 00:26:41 (1788841601) [ 3969.455034] Lustre: 110001:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 278, rollback = 2 [ 3969.462263] Lustre: 110001:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 92 previous similar messages [ 3969.468252] Lustre: 110001:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 3969.474908] Lustre: 110001:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 92 previous similar messages [ 3969.483893] Lustre: 110001:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3969.490341] Lustre: 110001:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 92 previous similar messages [ 3969.497357] Lustre: 110001:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3969.508989] Lustre: 110001:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 92 previous similar messages [ 3969.514893] Lustre: 110001:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 3969.523475] Lustre: 110001:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 92 previous similar messages [ 3969.529689] Lustre: 110001:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3969.535876] Lustre: 110001:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 92 previous similar messages [ 3970.788279] Lustre: *** cfs_fail_loc=1614, val=0*** [ 3971.306334] Lustre: *** cfs_fail_loc=1614, val=103*** [ 3971.311551] Lustre: Skipped 1 previous similar message [ 3975.352268] Lustre: 111921:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 275, rollback = 2 [ 3975.363402] Lustre: 111921:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 11 previous similar messages [ 3975.383135] Lustre: 111921:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3975.391027] Lustre: 111921:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3975.399902] Lustre: 111921:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 3/275/0 [ 3975.405646] Lustre: 111921:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3975.409722] Lustre: 111921:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 1/4/0, quota 7/225/0 [ 3975.414409] Lustre: 111921:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3975.421893] Lustre: 111921:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 3975.429475] Lustre: 111921:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3975.444533] Lustre: 111921:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3975.456888] Lustre: 111921:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3983.620594] Lustre: DEBUG MARKER: == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 00:26:56 (1788841616) [ 3984.025663] Lustre: 111937:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 3984.034476] Lustre: 111937:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 3984.041755] Lustre: 111937:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3984.047933] Lustre: 111937:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3984.053459] Lustre: 111937:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3984.061753] Lustre: 111937:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3984.068931] Lustre: 111937:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 3984.074950] Lustre: 111937:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3984.082728] Lustre: 111937:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 3984.089437] Lustre: 111937:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3984.093976] Lustre: 111937:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3984.101287] Lustre: 111937:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3987.390829] Lustre: *** cfs_fail_loc=1615, val=0*** [ 3987.396288] Lustre: Skipped 1 previous similar message [ 3989.021649] Lustre: *** cfs_fail_loc=1615, val=0*** [ 3989.031973] Lustre: Skipped 1 previous similar message [ 4011.969738] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_18b skipping excluded test 18b [ 4013.858353] Lustre: DEBUG MARKER: == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 00:27:27 (1788841647) [ 4014.292513] Lustre: 110000:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 4014.297736] Lustre: 110000:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 21 previous similar messages [ 4014.302168] Lustre: 110000:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 4014.306537] Lustre: 110000:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4014.310046] Lustre: 110000:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4014.314134] Lustre: 110000:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4014.318627] Lustre: 110000:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4014.321991] Lustre: 110000:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4014.325774] Lustre: 110000:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 4014.330079] Lustre: 110000:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4014.340752] Lustre: 110000:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4014.345437] Lustre: 110000:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4016.976416] Lustre: *** cfs_fail_loc=1617, val=0*** [ 4017.124731] Lustre: *** cfs_fail_loc=1617, val=0*** [ 4019.983106] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4019.989359] Lustre: Skipped 3 previous similar messages [ 4040.181947] Lustre: DEBUG MARKER: == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 00:27:53 (1788841673) [ 4042.519368] Lustre: *** cfs_fail_loc=1618, val=0*** [ 4042.527812] Lustre: Skipped 5 previous similar messages [ 4076.002068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4076.002648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4076.019703] Lustre: Skipped 1 previous similar message [ 4076.020204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4081.128704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4081.136616] Lustre: Skipped 6 previous similar messages [ 4082.274521] Lustre: server umount lustre-MDT0000 complete [ 4086.501360] LustreError: 109985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841722 with bad export cookie 11324526969413903084 [ 4086.508158] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4086.513048] LustreError: 109985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4086.975728] Lustre: server umount lustre-MDT0001 complete [ 4101.742059] Lustre: server umount lustre-OST0000 complete [ 4115.863420] Lustre: server umount lustre-OST0001 complete [ 4132.098064] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 4142.447629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4142.970476] LustreError: 118669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.013200] LustreError: 118669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4143.077880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4148.200498] LustreError: 118670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4148.231618] LustreError: 118670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4148.549140] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4152.317929] LustreError: 118669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4157.411874] LustreError: 118670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.487276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.035187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4165.123660] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4169.064347] Lustre: 119809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4177.267406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4177.686647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4184.870522] LustreError: 120162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4184.906214] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:33) [ 4185.157273] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4187.952600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:116 to 0x280000401:161) [ 4193.251441] LustreError: 120508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4193.281955] LustreError: 120508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4195.725729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4201.472618] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 4201.479383] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:33) [ 4203.208624] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4211.910601] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4215.926606] Lustre: 121680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4230.114656] Lustre: DEBUG MARKER: == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 00:31:03 (1788841863) [ 4230.448032] Lustre: 118664:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 4230.453835] Lustre: 118664:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 46 previous similar messages [ 4230.458986] Lustre: 118664:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4230.469100] Lustre: 118664:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4230.475722] Lustre: 118664:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4230.482836] Lustre: 118664:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4230.488910] Lustre: 118664:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 4230.496144] Lustre: 118664:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4230.502939] Lustre: 118664:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 4230.507230] Lustre: 118664:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4230.513614] Lustre: 118664:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4230.518491] Lustre: 118664:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4232.266140] Lustre: *** cfs_fail_loc=1618, val=0*** [ 4232.271209] Lustre: Skipped 3 previous similar messages [ 4267.489118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4267.500467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4267.529733] Lustre: Skipped 2 previous similar messages [ 4267.540818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4272.806462] Lustre: server umount lustre-MDT0000 complete [ 4273.123762] LustreError: 118665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4273.154235] LustreError: 118665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4276.942118] LustreError: 118650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841912 with bad export cookie 11324526969413918344 [ 4276.949803] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4276.956201] LustreError: 118650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4277.262481] Lustre: server umount lustre-MDT0001 complete [ 4291.565780] Lustre: server umount lustre-OST0000 complete [ 4294.444155] Lustre: 107154:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841914/real 1788841914] req@ffff9d16ff61fb80 x1875736175865856/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4294.494162] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.520138] Lustre: Skipped 3 previous similar messages [ 4296.263530] Lustre: server umount lustre-OST0001 complete [ 4312.639948] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 4323.111189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.545458] LustreError: 124249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4323.632947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.639651] Lustre: Skipped 1 previous similar message [ 4328.176754] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4336.556989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.691553] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4344.530829] Lustre: 125390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4350.980482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4357.963603] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4361.530378] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:65) [ 4366.949662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4368.247942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4368.254951] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 4368.262801] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:65) [ 4374.085599] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4381.409523] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4384.868417] Lustre: 127261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4391.923826] Lustre: 124982:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 252 < left 260, rollback = 2 [ 4391.949881] Lustre: 124982:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 21 previous similar messages [ 4391.963674] Lustre: 124982:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4391.986724] Lustre: 124982:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4392.004730] Lustre: 124982:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/260/0 [ 4392.017335] Lustre: 124982:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4392.025634] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4392.034222] Lustre: 124982:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 0/0/0, quota 4/150/2 [ 4392.034232] Lustre: 124982:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4392.034239] Lustre: 124982:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 1/17/2, delete: 0/0/0 [ 4392.034243] Lustre: 124982:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4392.034249] Lustre: 124982:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4392.034252] Lustre: 124982:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4392.145856] Lustre: Skipped 1 previous similar message [ 4419.191365] Lustre: DEBUG MARKER: == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 00:34:12 (1788842052) [ 4421.921189] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4421.927978] Lustre: Skipped 3 previous similar messages [ 4428.397282] Lustre: *** cfs_fail_loc=161c, val=0*** [ 4447.424972] Lustre: DEBUG MARKER: == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) ========================================================== 00:34:41 (1788842081) [ 4450.052828] Lustre: *** cfs_fail_loc=162e, val=0*** [ 4462.869533] Lustre: DEBUG MARKER: == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 00:34:56 (1788842096) [ 4467.360710] Lustre: *** cfs_fail_loc=162f, val=0*** [ 4467.365583] Lustre: Skipped 9 previous similar messages [ 4483.478926] Lustre: DEBUG MARKER: == sanity-lfsck test 19a: OST-object inconsistency self detect ========================================================== 00:35:17 (1788842117) [ 4495.202213] Lustre: DEBUG MARKER: == sanity-lfsck test 19b: OST-object inconsistency self repair ========================================================== 00:35:28 (1788842128) [ 4498.279569] Lustre: *** cfs_fail_loc=1611, val=0*** [ 4498.302108] Lustre: *** cfs_fail_loc=1611, val=0*** [ 4498.306631] Lustre: Skipped 3 previous similar messages [ 4503.421260] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.10@tcp inode [0x2000013a1:0x1f:0x0] object 0x280000401:203 extent [0-4095], client returned csum 0 (type 4), server csum 703fb3ad (type 4) [ 4504.507790] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.10@tcp inode [0x2000013a1:0x20:0x0] object 0x280000401:204 extent [0-4095], client returned csum 0 (type 4), server csum 7cbb935c (type 4) [ 4511.792924] Lustre: DEBUG MARKER: == sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================================================== 00:35:45 (1788842145) [ 4522.039448] Lustre: 131720:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 263, rollback = 2 [ 4522.053942] Lustre: 131720:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 125 previous similar messages [ 4522.078712] Lustre: 131720:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4522.094467] Lustre: 131720:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4522.106599] Lustre: 131720:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 2/263/0 [ 4522.116537] Lustre: 131720:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4522.130313] Lustre: 131720:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/2 [ 4522.138015] Lustre: 131720:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4522.149564] Lustre: 131720:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 4522.155451] Lustre: 131720:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4522.163447] Lustre: 131720:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4522.177326] Lustre: 131720:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4536.998108] Lustre: DEBUG MARKER: == sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case ========================================================== 00:36:10 (1788842170) [ 4543.419277] Lustre: DEBUG MARKER: == sanity-lfsck test 21: run all LFSCK components by default ========================================================== 00:36:16 (1788842176) [ 4557.738438] Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) ========================================================== 00:36:31 (1788842191) [ 4560.116803] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4560.126560] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4560.135249] Lustre: Skipped 1 previous similar message [ 4572.184745] Lustre: DEBUG MARKER: == sanity-lfsck test 22b: LFSCK can repair unmatched pairs (2) ========================================================== 00:36:45 (1788842205) [ 4573.847085] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4573.854218] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4584.519294] Lustre: DEBUG MARKER: == sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) ========================================================== 00:36:58 (1788842218) [ 4586.373546] Lustre: *** cfs_fail_loc=1620, val=0*** [ 4602.692321] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_23b skipping excluded test 23b [ 4604.569562] Lustre: DEBUG MARKER: == sanity-lfsck test 23c: LFSCK can repair dangling name entry (3) ========================================================== 00:37:18 (1788842238) [ 4612.020306] Lustre: *** cfs_fail_loc=1621, val=127*** [ 4612.026993] Lustre: Skipped 1 previous similar message [ 4615.672589] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4639.108646] Lustre: DEBUG MARKER: == sanity-lfsck test 23d: LFSCK can repair a dangling name entry to a remote object ========================================================== 00:37:51 (1788842271) [ 4641.736160] Lustre: Failing over lustre-MDT0000 [ 4642.190808] Lustre: server umount lustre-MDT0000 complete [ 4642.795635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4642.803137] LustreError: 125764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.810936] Lustre: Skipped 1 previous similar message [ 4642.870773] LustreError: 125764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4653.716070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4653.923690] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4654.166443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.169964] Lustre: Skipped 3 previous similar messages [ 4654.217267] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4655.107481] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4659.208076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4659.246860] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4659.309245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 4659.309624] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:130 to 0x2c0000401:161) [ 4659.991538] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4662.151708] LustreError: 132569:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0000: [0x2000013a3:0x82:0x0] doesn't exist!: rc = -14 [ 4673.760661] Lustre: DEBUG MARKER: == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ========================================================== 00:38:27 (1788842307) [ 4676.289800] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4676.511488] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4676.520574] Lustre: Skipped 1 previous similar message [ 4689.157165] Lustre: DEBUG MARKER: == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ========================================================== 00:38:42 (1788842322) [ 4691.138308] Lustre: *** cfs_fail_loc=1623, val=0*** [ 4703.290256] Lustre: DEBUG MARKER: == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace ========================================================== 00:38:56 (1788842336) [ 4704.973713] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4717.445119] Lustre: DEBUG MARKER: == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========================================================== 00:39:10 (1788842350) [ 4731.335143] Lustre: DEBUG MARKER: == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan ========================================================== 00:39:24 (1788842364) [ 4733.339768] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4733.345689] Lustre: Skipped 1 previous similar message [ 4745.966491] Lustre: DEBUG MARKER: == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ========================================================== 00:39:39 (1788842379) [ 4760.905349] Lustre: DEBUG MARKER: == sanity-lfsck test 28: Skip the failed MDT(s) when handle orphan MDT-objects ========================================================== 00:39:54 (1788842394) [ 4768.350982] Lustre: *** cfs_fail_loc=161c, val=0*** [ 4768.353603] Lustre: Skipped 1 previous similar message [ 4786.855638] Lustre: DEBUG MARKER: == sanity-lfsck test 29b: LFSCK can repair bad nlink count (2) ========================================================== 00:40:20 (1788842420) [ 4787.459722] Lustre: 124244:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 4787.465740] Lustre: 124244:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 497 previous similar messages [ 4787.469787] Lustre: 124244:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 4787.475188] Lustre: 124244:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4787.479585] Lustre: 124244:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4787.484709] Lustre: 124244:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4787.489365] Lustre: 124244:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4787.494581] Lustre: 124244:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4787.498326] Lustre: 124244:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 4787.503317] Lustre: 124244:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4787.508126] Lustre: 124244:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4787.513342] Lustre: 124244:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4789.034399] Lustre: *** cfs_fail_loc=1626, val=0*** [ 4789.046185] Lustre: Skipped 4 previous similar messages [ 4800.085472] Lustre: DEBUG MARKER: == sanity-lfsck test 29c: verify linkEA size limitation == 00:40:33 (1788842433) [ 4829.613403] Lustre: DEBUG MARKER: == sanity-lfsck test 29d: accessing non-existing inode shouldn't turn fs read-only (ldiskfs) ========================================================== 00:41:03 (1788842463) [ 4832.199692] LustreError: 125764:0:(osd_handler.c:284:osd_idc_find_or_init()) lustre-MDT0000: cannot lookup FID [0x2000013a3:0x9e:0x0]: rc = -2 [ 4837.743182] Lustre: DEBUG MARKER: == sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 00:41:11 (1788842471) [ 4863.148092] Lustre: Failing over lustre-MDT0000 [ 4863.461901] Lustre: server umount lustre-MDT0000 complete [ 4863.977498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4863.984523] LustreError: 140869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.986877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4864.002463] Lustre: Skipped 5 previous similar messages [ 4864.031640] LustreError: 140869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 4874.184509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4874.368841] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4874.570313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4874.627607] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4879.316211] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4879.849127] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4879.850299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4879.866304] Lustre: Skipped 3 previous similar messages [ 4879.945518] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4879.988691] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:167 to 0x2c0000401:193) [ 4879.990521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:294 to 0x280000401:321) [ 4892.875629] Lustre: DEBUG MARKER: == sanity-lfsck test 31a: The LFSCK can find/repair the name entry with bad name hash (1) ========================================================== 00:42:06 (1788842526) [ 4906.958979] Lustre: DEBUG MARKER: == sanity-lfsck test 31b: The LFSCK can find/repair the name entry with bad name hash (2) ========================================================== 00:42:20 (1788842540) [ 4921.023146] Lustre: DEBUG MARKER: == sanity-lfsck test 31c: Re-generate the lost master LMV EA for striped directory ========================================================== 00:42:34 (1788842554) [ 4922.565246] Lustre: *** cfs_fail_loc=1629, val=0*** [ 4922.575813] Lustre: Skipped 7 previous similar messages [ 4934.829292] Lustre: DEBUG MARKER: == sanity-lfsck test 31d: Set broken striped directory (modified after broken) as read-only ========================================================== 00:42:48 (1788842568) [ 4940.997504] Lustre: Failing over lustre-MDT0000 [ 4941.283522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4941.293406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4941.412184] Lustre: server umount lustre-MDT0000 complete [ 4950.732320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.920676] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4951.188295] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4955.917332] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4956.656139] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4956.656186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4956.674904] Lustre: Skipped 3 previous similar messages [ 4956.702890] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4956.763896] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:167 to 0x2c0000401:225) [ 4956.768169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:294 to 0x280000401:353) [ 4964.701326] Lustre: Failing over lustre-MDT0000 [ 4966.885953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4966.893382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4966.901873] Lustre: Skipped 3 previous similar messages [ 4966.973649] Lustre: server umount lustre-MDT0000 complete [ 4976.006974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4976.148131] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4976.390986] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4978.699280] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4981.740677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4981.746283] Lustre: Skipped 3 previous similar messages [ 4981.779989] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4981.791414] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4981.831581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4981.831581] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:167 to 0x2c0000401:257) [ 4990.815874] Lustre: DEBUG MARKER: == sanity-lfsck test 31e: Re-generate the lost slave LMV EA for striped directory (1) ========================================================== 00:43:44 (1788842624) [ 5004.985516] Lustre: DEBUG MARKER: == sanity-lfsck test 31f: Re-generate the lost slave LMV EA for striped directory (2) ========================================================== 00:43:58 (1788842638) [ 5019.590643] Lustre: DEBUG MARKER: == sanity-lfsck test 31g: Repair the corrupted slave LMV EA ========================================================== 00:44:12 (1788842652) [ 5060.046700] Lustre: DEBUG MARKER: == sanity-lfsck test 31h: Repair the corrupted shard's name entry ========================================================== 00:44:53 (1788842693) [ 5061.642411] Lustre: *** cfs_fail_loc=162c, val=0*** [ 5061.645872] Lustre: Skipped 13 previous similar messages [ 5077.504189] Lustre: DEBUG MARKER: == sanity-lfsck test 32a: stop LFSCK when some OST failed ========================================================== 00:45:10 (1788842710) [ 5086.887452] LustreError: 148557:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5089.791690] Lustre: Failing over lustre-OST0000 [ 5089.939802] Lustre: server umount lustre-OST0000 complete [ 5089.976344] LustreError: 148557:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 5089.993819] LustreError: 148557:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5092.657098] LustreError: 148557:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout interrupted [ 5092.672550] LustreError: lustre-OST0000-osc-MDT0000: operation lfsck_notify to node 0@lo failed: rc = -107 [ 5092.677739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.686826] Lustre: Skipped 3 previous similar messages [ 5104.161628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5104.492206] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5106.413764] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5106.435111] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5106.435291] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5106.460271] Lustre: Skipped 3 previous similar messages [ 5111.337390] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5122.128542] Lustre: DEBUG MARKER: == sanity-lfsck test 32b: stop LFSCK when some MDT failed ========================================================== 00:45:55 (1788842755) [ 5136.678885] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 5156.082408] Lustre: 151362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5180.621978] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5185.015718] Lustre: 152496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5199.191911] LustreError: 152635:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5202.256402] LustreError: 152635:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 5202.263681] LustreError: 152635:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5202.270118] LustreError: 152635:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) Skipped 1 previous similar message [ 5202.481074] Lustre: Failing over lustre-MDT0001 [ 5205.306064] LustreError: 152634:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 5205.320997] LustreError: 152634:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) Skipped 1 previous similar message [ 5205.336631] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 5205.358811] LustreError: Skipped 1 previous similar message [ 5205.377159] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5205.398774] Lustre: Skipped 1 previous similar message [ 5205.412970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5205.424585] Lustre: Skipped 4 previous similar messages [ 5205.738885] Lustre: server umount lustre-MDT0001 complete [ 5207.011522] LustreError: 132569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5207.039897] LustreError: 132569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 34 previous similar messages [ 5220.076744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5220.407361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5220.422613] Lustre: Skipped 3 previous similar messages [ 5220.455258] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5224.923457] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5225.448510] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5225.454292] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5225.475814] Lustre: Skipped 1 previous similar message [ 5225.499929] Lustre: lustre-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5225.554472] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:97) [ 5225.554773] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:97) [ 5233.716894] Lustre: DEBUG MARKER: == sanity-lfsck test 33: check LFSCK paramters =========== 00:47:47 (1788842867) [ 5251.546213] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 5273.262326] Lustre: 155356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5298.836545] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5302.568635] Lustre: 156492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5305.450923] Lustre: 124245:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 5305.459828] Lustre: 124245:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1220 previous similar messages [ 5305.465973] Lustre: 124245:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 5305.470183] Lustre: 124245:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1220 previous similar messages [ 5305.476281] Lustre: 124245:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5305.485919] Lustre: 124245:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1220 previous similar messages [ 5305.491620] Lustre: 124245:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 5305.498084] Lustre: 124245:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1220 previous similar messages [ 5305.503531] Lustre: 124245:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 5305.508650] Lustre: 124245:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1220 previous similar messages [ 5305.514441] Lustre: 124245:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5305.519121] Lustre: 124245:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1220 previous similar messages [ 5324.435678] Lustre: DEBUG MARKER: == sanity-lfsck test 34: LFSCK can rebuild the lost agent object ========================================================== 00:49:17 (1788842957) [ 5326.045308] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_34 Only valid for ZFS backend [ 5327.547786] Lustre: DEBUG MARKER: == sanity-lfsck test 35: LFSCK can rebuild the lost agent entry ========================================================== 00:49:21 (1788842961) [ 5335.583533] Lustre: *** cfs_fail_loc=1631, val=0*** [ 5348.323090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5348.323579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5348.332586] Lustre: Skipped 4 previous similar messages [ 5348.332752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5348.350354] Lustre: Skipped 3 previous similar messages [ 5352.564095] Lustre: server umount lustre-MDT0000 complete [ 5356.470297] LustreError: 130206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842992 with bad export cookie 11324526969413991151 [ 5356.474306] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5356.480487] LustreError: 130206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5356.732314] Lustre: server umount lustre-MDT0001 complete [ 5371.891132] Lustre: server umount lustre-OST0000 complete [ 5374.944843] Lustre: 107153:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788842994/real 1788842994] req@ffff9d16cfd94a80 x1875736177265920/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843010 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5377.919400] Lustre: server umount lustre-OST0001 complete [ 5398.964435] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 5408.785983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5414.459355] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5424.225396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5429.399811] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5432.342834] Lustre: 160385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5438.812795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5444.784306] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5449.323931] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:129) [ 5454.017226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5455.233390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:540 to 0x280000401:577) [ 5455.239410] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:412 to 0x2c0000401:449) [ 5455.346220] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:129) [ 5460.116561] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5467.604522] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5471.234569] Lustre: 162256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5480.125474] Lustre: DEBUG MARKER: == sanity-lfsck test 36a: rebuild LOV EA for mirrored file (1) ========================================================== 00:51:53 (1788843113) [ 5481.874978] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36a needs >= 3 OSTs [ 5483.473936] Lustre: DEBUG MARKER: == sanity-lfsck test 36b: rebuild LOV EA for mirrored file (2) ========================================================== 00:51:57 (1788843117) [ 5485.228295] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36b needs >= 3 OSTs [ 5487.082541] Lustre: DEBUG MARKER: == sanity-lfsck test 36c: rebuild LOV EA for mirrored file (3) ========================================================== 00:52:00 (1788843120) [ 5488.671711] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36c needs >= 3 OSTs [ 5490.692293] Lustre: DEBUG MARKER: == sanity-lfsck test 37: LFSCK must skip a ORPHAN ======== 00:52:04 (1788843124) [ 5503.160276] Lustre: DEBUG MARKER: == sanity-lfsck test 38: LFSCK does not break foreign file and reverse is also true ========================================================== 00:52:16 (1788843136) [ 5519.860499] Lustre: DEBUG MARKER: == sanity-lfsck test 39: LFSCK does not break foreign dir and reverse is also true ========================================================== 00:52:33 (1788843153) [ 5535.052828] Lustre: DEBUG MARKER: == sanity-lfsck test 40a: LFSCK correctly fixes lmm_oi in composite layout ========================================================== 00:52:48 (1788843168) [ 5550.950678] Lustre: DEBUG MARKER: == sanity-lfsck test 41: SEL support in LFSCK ============ 00:53:04 (1788843184) [ 5571.988195] Lustre: DEBUG MARKER: == sanity-lfsck test 42: LFSCK repairs inconsistent MDT-object/OST-object encryption flags ========================================================== 00:53:25 (1788843205) [ 5606.797148] Lustre: *** cfs_fail_loc=1632, val=0*** [ 5625.347685] Lustre: DEBUG MARKER: == sanity-lfsck test 43: LFSCK does not loop endlessly on iget failure in scanning-phase1 ========================================================== 00:54:18 (1788843258) [ 5628.597449] Lustre: Failing over lustre-MDT0001 [ 5628.937488] Lustre: server umount lustre-MDT0001 complete [ 5629.412517] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5629.432896] Lustre: Skipped 3 previous similar messages [ 5636.801483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5637.110793] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5637.110928] Lustre: lustre-MDT0001: Aborting client recovery [ 5637.122472] LustreError: 166049:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0001: Aborting recovery [ 5637.129086] Lustre: 166073:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5637.139854] Lustre: 166073:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client c2ccdaba-a3df-4627-ba75-fea384829dab@ [ 5637.152482] Lustre: lustre-MDT0001: disconnecting 2 stale clients [ 5637.160212] Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] [ 5637.168551] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] [ 5637.214481] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:161) [ 5637.216375] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:131 to 0x2c0000400:161) [ 5642.216464] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5642.246985] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5642.259340] Lustre: Skipped 4 previous similar messages [ 5642.556712] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5649.560845] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid [ 5649.807087] Lustre: DEBUG MARKER: osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid in FULL state after 0 sec [ 5657.127988] Lustre: DEBUG MARKER: == sanity-lfsck test 44: umount while lfsck is stopping == 00:54:50 (1788843290) [ 5666.451716] Lustre: *** cfs_fail_loc=1600, val=3*** [ 5670.119287] Lustre: Failing over lustre-MDT0000 [ 5670.473693] Lustre: server umount lustre-MDT0000 complete [ 5672.932454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5680.569673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.650793] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5681.106659] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5681.128262] Lustre: Skipped 2 previous similar messages [ 5682.204653] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5685.944509] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5686.250907] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5686.306792] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5686.339786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:621 to 0x280000401:641) [ 5686.340132] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 5695.545707] Lustre: DEBUG MARKER: == sanity-lfsck test 45: LFSCK should fix UID/GID/PROJID of OST object ========================================================== 00:55:29 (1788843329) [ 5740.184508] Lustre: Failing over lustre-OST0000 [ 5740.287177] Lustre: server umount lustre-OST0000 complete [ 5742.564239] LustreError: 160738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.585427] LustreError: 160738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 33 previous similar messages [ 5746.599456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5758.466500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5758.637745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5758.652994] Lustre: Skipped 6 previous similar messages [ 5758.670936] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5758.980817] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5760.236160] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 5760.240037] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5760.260084] Lustre: Skipped 3 previous similar messages [ 5764.819587] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5771.834653] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5772.162333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5776.499895] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5776.681895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5783.272464] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a1e18b79000.ost_server_uuid 50 [ 5785.066833] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a1e18b79000.ost_server_uuid in FULL state after 0 sec [ 5870.561137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5870.573125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5876.713834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5876.726096] Lustre: Skipped 7 previous similar messages [ 5876.897219] Lustre: server umount lustre-MDT0000 complete [ 5886.045159] LustreError: 159225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843521 with bad export cookie 11324526969414074017 [ 5886.055121] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5886.068988] LustreError: 159225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5886.363813] Lustre: server umount lustre-MDT0001 complete [ 5902.178657] Lustre: 107153:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843522/real 1788843522] req@ffff9d16d0f65500 x1875736177675136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843538 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5905.519254] Lustre: server umount lustre-OST0000 complete [ 5907.173631] Lustre: 107152:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843527/real 1788843527] req@ffff9d15d02ad500 x1875736177675520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5913.064159] Lustre: 107154:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843532/real 1788843532] req@ffff9d15c1e8df80 x1875736177676032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5914.630440] Lustre: server umount lustre-OST0001 complete [ 5932.493808] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 5935.379924] Key type lgssc unregistered [ 5935.705267] LNet: 175744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5935.716978] LNetError: 175744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5935.733270] LNet: Removed LNI 192.168.203.110@tcp [ 5936.668150] Key type .llcrypt unregistered [ 5936.673431] Key type ._llcrypt unregistered [ 5962.338806] Key type ._llcrypt registered [ 5962.341931] Key type .llcrypt registered [ 5962.493132] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 5981.434790] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 5982.767368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5982.995896] alg: No test for adler32 (adler32-zlib) [ 5984.134212] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 5984.469455] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 5986.144222] Key type lgssc registered [ 5987.478284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6040.560446] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 6054.844314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6054.900486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.294264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6056.340077] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6056.440708] Lustre: lustre-MDT0000: new disk, initializing [ 6056.518225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6056.533603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6061.499534] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6074.838170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6074.972955] Lustre: 180206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6075.021854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6075.027155] Lustre: Skipped 1 previous similar message [ 6075.118882] Lustre: lustre-MDT0001: new disk, initializing [ 6075.196449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6075.227541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6075.235986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6080.280149] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6085.000221] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6094.533320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6094.824089] Lustre: lustre-OST0000: new disk, initializing [ 6094.828530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6094.834686] Lustre: 182145:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6094.884733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.602100] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6102.580630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6102.593934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6102.664871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6114.064866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6114.184738] Lustre: lustre-OST0001: new disk, initializing [ 6114.188555] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6114.197135] Lustre: 183168:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6114.264557] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6120.486644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6120.496336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6120.531569] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6120.594205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6133.360907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6141.885069] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6148.388772] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 01:03:01 (1788843781) === [ 6150.235636] Lustre: DEBUG MARKER: == sanity-lfsck test complete, duration 5821 sec ========= 01:03:03 (1788843783) [ 6152.326544] Lustre: DEBUG MARKER: === sanity-lfsck: start cleanup 01:03:05 (1788843785) === [ 6156.010818] Lustre: DEBUG MARKER: === sanity-lfsck: finish cleanup 01:03:09 (1788843789) === [ 6161.377760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6161.384089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6161.410526] Lustre: Skipped 3 previous similar messages [ 6161.425607] Lustre: Skipped 3 previous similar messages [ 6166.498446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6166.507468] Lustre: Skipped 3 previous similar messages [ 6167.781140] Lustre: server umount lustre-MDT0000 complete [ 6176.739874] LustreError: 180214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6176.775428] LustreError: 180214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6178.872944] LustreError: 182143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843814 with bad export cookie 13670087193317164298 [ 6178.874486] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6178.891269] LustreError: 182143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6179.295488] Lustre: server umount lustre-MDT0001 complete [ 6197.216841] Lustre: 177354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843817/real 1788843817] req@ffff9d16ee8eb100 x1875738487771008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6197.241752] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6198.420065] Lustre: server umount lustre-OST0000 complete [ 6199.457079] Lustre: 177357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843819/real 1788843819] req@ffff9d16f6aebb80 x1875738487771264/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6202.339224] Lustre: 177355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843822/real 1788843822] req@ffff9d15ced0b800 x1875738487771520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6205.668931] Lustre: 177357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843825/real 1788843825] req@ffff9d16e9204000 x1875738487771904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6207.241418] Lustre: server umount lustre-OST0001 complete [ 6223.961687] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 6227.004552] Key type lgssc unregistered [ 6227.349459] LNet: 186644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6227.356527] LNetError: 186644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6227.382059] LNet: Removed LNI 192.168.203.110@tcp [ 6228.460163] Key type .llcrypt unregistered [ 6228.461898] Key type ._llcrypt unregistered