[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 575205616 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] BRK [0x48601000, 0x48601fff] PGTABLE [ 0.000000] BRK [0x48602000, 0x48602fff] PGTABLE [ 0.000000] BRK [0x48603000, 0x48603fff] PGTABLE [ 0.000000] BRK [0x48604000, 0x48604fff] PGTABLE [ 0.000000] BRK [0x48605000, 0x48605fff] PGTABLE [ 0.000000] BRK [0x48606000, 0x48606fff] PGTABLE [ 0.000000] BRK [0x48607000, 0x48607fff] PGTABLE [ 0.000000] BRK [0x48608000, 0x48608fff] PGTABLE [ 0.000000] BRK [0x48609000, 0x48609fff] PGTABLE [ 0.000000] BRK [0x4860a000, 0x4860afff] PGTABLE [ 0.000000] BRK [0x4860b000, 0x4860bfff] PGTABLE [ 0.000000] BRK [0x4860c000, 0x4860cfff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] On node 0 totalpages: 1076588 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 158 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:63 [ 0.000000] Normal zone: 4536 pages used for memmap [ 0.000000] Normal zone: 290304 pages, LIFO batch:63 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] pcpu-alloc: s221184 r8192 d28672 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001000] hpet clockevent registered [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002396] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.007027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009148] LSM: Security Framework initializing [ 0.010064] Yama: becoming mindful. [ 0.011049] SELinux: Initializing. [ 0.012071] *** VALIDATE selinux *** [ 0.020795] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026494] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028024] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030025] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032097] *** VALIDATE tmpfs *** [ 0.034212] *** VALIDATE proc *** [ 0.035262] *** VALIDATE cgroup *** [ 0.036012] *** VALIDATE cgroup2 *** [ 0.038104] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041035] Spectre V2 : User space: Vulnerable [ 0.042011] Speculative Store Bypass: Vulnerable [ 0.045840] debug: unmapping init [mem 0xffffffffa2859000-0xffffffffa2860fff] [ 0.048197] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050355] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051027] ... version: 2 [ 0.052013] ... bit width: 48 [ 0.053013] ... generic registers: 4 [ 0.054013] ... value mask: 0000ffffffffffff [ 0.055018] ... max period: 00007fffffffffff [ 0.056023] ... fixed-purpose events: 3 [ 0.057012] ... event mask: 000000070000000f [ 0.058328] rcu: Hierarchical SRCU implementation. [ 0.061193] smp: Bringing up secondary CPUs ... [ 0.062984] x86: Booting SMP configuration: [ 0.063037] .... node #0, CPUs: #1 #2 #3 [ 0.068100] smp: Brought up 1 node, 4 CPUs [ 0.070015] smpboot: Max logical packages: 1 [ 0.071013] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.165024] node 0 deferred pages initialised in 92ms [ 0.170109] devtmpfs: initialized [ 0.172553] x86/mm: Memory block size: 128MB [ 0.175000] gcov: version magic: 0x41383552 [ 0.179378] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.182290] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.186592] pinctrl core: initialized pinctrl subsystem [ 0.189008] ************************************************************* [ 0.192012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196021] ** ** [ 0.198007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.201011] ** ** [ 0.204016] ** This means that this kernel is built to expose internal ** [ 0.206009] ** IOMMU data structures, which may compromise security on ** [ 0.209011] ** your system. ** [ 0.212012] ** ** [ 0.214010] ** If you see this message and you are not debugging the ** [ 0.217025] ** kernel, report this immediately to your vendor! ** [ 0.219011] ** ** [ 0.221012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.224098] ************************************************************* [ 0.228924] NET: Registered protocol family 16 [ 0.231554] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.234068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.240287] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.245020] cpuidle: using governor menu [ 0.247478] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.250433] PCI: Using configuration type 1 for base access [ 0.252129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.262109] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.266084] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.269176] cryptd: max_cpu_qlen set to 1000 [ 0.273029] ACPI: Added _OSI(Module Device) [ 0.275011] ACPI: Added _OSI(Processor Device) [ 0.276011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.278014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.282256] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.288388] ACPI: Interpreter enabled [ 0.290009] ACPI: PM: (supports S0 S3 S4 S5) [ 0.292015] ACPI: Using IOAPIC for interrupt routing [ 0.295118] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.299437] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.312000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.315049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.317021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.322097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.329225] acpiphp: Slot [2] registered [ 0.332167] acpiphp: Slot [5] registered [ 0.334122] acpiphp: Slot [6] registered [ 0.337164] acpiphp: Slot [7] registered [ 0.339222] acpiphp: Slot [8] registered [ 0.341405] acpiphp: Slot [9] registered [ 0.343149] acpiphp: Slot [10] registered [ 0.345136] acpiphp: Slot [3] registered [ 0.347102] acpiphp: Slot [4] registered [ 0.349118] acpiphp: Slot [11] registered [ 0.351123] acpiphp: Slot [12] registered [ 0.354128] acpiphp: Slot [13] registered [ 0.356132] acpiphp: Slot [14] registered [ 0.358225] acpiphp: Slot [15] registered [ 0.360440] acpiphp: Slot [16] registered [ 0.363281] acpiphp: Slot [17] registered [ 0.364092] acpiphp: Slot [18] registered [ 0.365091] acpiphp: Slot [19] registered [ 0.367114] acpiphp: Slot [20] registered [ 0.368106] acpiphp: Slot [21] registered [ 0.370114] acpiphp: Slot [22] registered [ 0.372121] acpiphp: Slot [23] registered [ 0.374140] acpiphp: Slot [24] registered [ 0.376118] acpiphp: Slot [25] registered [ 0.378113] acpiphp: Slot [26] registered [ 0.379396] acpiphp: Slot [27] registered [ 0.382125] acpiphp: Slot [28] registered [ 0.384108] acpiphp: Slot [29] registered [ 0.385133] acpiphp: Slot [30] registered [ 0.387107] acpiphp: Slot [31] registered [ 0.388060] PCI host bridge to bus 0000:00 [ 0.390023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.392024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.395021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.397026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.401030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.404199] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.407184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.410340] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.415064] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.427014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.433065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.436017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.440021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.443022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.447188] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.451117] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.455051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.459854] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.466013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.478000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.484011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.493954] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.503021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.513020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.534036] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.543402] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.550015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.556015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.574031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.585729] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.590016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.597017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.616017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.635396] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.650018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.658015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.675018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.688865] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.697014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.704020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.724020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.740000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.749017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.755015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.772017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.784598] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.788489] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.791508] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.794591] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.797226] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.805110] iommu: Default domain type: Passthrough [ 0.806505] SCSI subsystem initialized [ 0.808141] ACPI: bus type USB registered [ 0.809111] usbcore: registered new interface driver usbfs [ 0.811084] usbcore: registered new interface driver hub [ 0.813084] usbcore: registered new device driver usb [ 0.815316] pps_core: LinuxPPS API ver. 1 registered [ 0.816007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.818048] PTP clock support registered [ 0.820427] EDAC MC: Ver: 3.0.0 [ 0.822086] PCI: Using ACPI for IRQ routing [ 0.823293] PCI: pci_cache_line_size set to 64 bytes [ 0.823585] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.823599] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.823603] e820: reserve RAM buffer [mem 0x146e00000-0x147ffffff] [ 0.824338] NetLabel: Initializing [ 0.826015] NetLabel: domain hash size = 128 [ 0.827009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.830277] NetLabel: unlabeled traffic allowed by default [ 0.833151] vgaarb: loaded [ 0.834241] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.837014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.845410] clocksource: Switched to clocksource kvm-clock [ 0.976372] VFS: Disk quotas dquot_6.6.0 [ 0.978539] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.982478] *** VALIDATE ramfs *** [ 0.984611] *** VALIDATE hugetlbfs *** [ 0.987061] pnp: PnP ACPI init [ 0.988853] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.988933] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.988962] pnp 00:02: [dma 2] [ 0.989040] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.989109] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.989193] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.989256] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.990240] pnp: PnP ACPI: found 6 devices [ 1.019627] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.024023] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.027252] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.029372] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.032584] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.036519] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.040068] NET: Registered protocol family 2 [ 1.043199] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.049767] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.054764] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.062046] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.067636] TCP: Hash tables configured (established 65536 bind 65536) [ 1.071403] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.075336] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.078172] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.081443] NET: Registered protocol family 1 [ 1.085215] RPC: Registered named UNIX socket transport module. [ 1.087683] RPC: Registered udp transport module. [ 1.088933] RPC: Registered tcp transport module. [ 1.091043] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.094501] NET: Registered protocol family 44 [ 1.096418] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.099189] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.102329] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.104364] PCI: CLS 0 bytes, default 64 [ 1.106219] Unpacking initramfs... [ 2.796810] debug: unmapping init [mem 0xffffa0c0bcc54000-0xffffa0c0bffbffff] [ 2.801052] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.803681] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.806455] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 3.398978] Initialise system trusted keyrings [ 3.401665] Key type blacklist registered [ 3.404399] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.417501] zbud: loaded [ 3.447563] *** VALIDATE nfs *** [ 3.448991] *** VALIDATE nfs4 *** [ 3.454191] pstore: using deflate compression [ 3.458430] Platform Keyring initialized [ 3.608857] NET: Registered protocol family 38 [ 3.611178] Key type asymmetric registered [ 3.613028] Asymmetric key parser 'x509' registered [ 3.615740] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.619974] io scheduler mq-deadline registered [ 3.621842] io scheduler kyber registered [ 3.624332] io scheduler bfq registered [ 3.626797] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.630781] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.634385] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.637991] ACPI: Power Button [PWRF] [ 3.644731] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.652653] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.681198] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.694498] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.714358] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.745246] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.777141] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.783700] Non-volatile memory driver v1.3 [ 3.786479] Linux agpgart interface v0.103 [ 3.829811] virtio_blk virtio1: [vda] 134592 512-byte logical blocks (68.9 MB/65.7 MiB) [ 3.834284] vda: detected capacity change from 0 to 68911104 [ 3.856079] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.859980] vdb: detected capacity change from 0 to 1073741824 [ 3.887880] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.890862] vdc: detected capacity change from 0 to 2621440000 [ 3.911359] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.915601] vdd: detected capacity change from 0 to 2621440000 [ 3.947907] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.952387] vde: detected capacity change from 0 to 4294967296 [ 3.973360] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.977362] vdf: detected capacity change from 0 to 4294967296 [ 3.987232] libphy: Fixed MDIO Bus: probed [ 3.994696] usbcore: registered new interface driver usbserial_generic [ 3.999609] usbserial: USB Serial support registered for generic [ 4.003328] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.010356] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.013106] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.017140] mousedev: PS/2 mouse device common for all mice [ 4.020588] rtc_cmos 00:05: RTC can wake from S4 [ 4.026422] rtc_cmos 00:05: registered as rtc0 [ 4.028320] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.032431] intel_pstate: CPU model not supported [ 4.035417] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.038180] hid: raw HID events driver (C) Jiri Kosina [ 4.042748] usbcore: registered new interface driver usbhid [ 4.044279] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.046788] usbhid: USB HID core driver [ 4.054811] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.054970] drop_monitor: Initializing network drop monitor service [ 4.063819] Initializing XFRM netlink socket [ 4.067329] NET: Registered protocol family 10 [ 4.071415] Segment Routing with IPv6 [ 4.073372] NET: Registered protocol family 17 [ 4.077210] mpls_gso: MPLS GSO support [ 4.078708] start plist test [ 4.082711] end plist test [ 4.084217] RAS: Correctable Errors collector initialized. [ 4.086983] AVX version of gcm_enc/dec engaged. [ 4.089072] AES CTR mode by8 optimization enabled [ 4.202528] sched_clock: Marking stable (4202499260, 0)->(5299859772, -1097360512) [ 4.207067] registered taskstats version 1 [ 4.212210] Loading compiled-in X.509 certificates [ 4.215216] zswap: loaded using pool lzo/zbud [ 4.253648] Key type big_key registered [ 4.276106] Key type encrypted registered [ 4.277830] ima: No TPM chip found, activating TPM-bypass! [ 4.280650] ima: Allocated hash algorithm: sha1 [ 4.283225] ima: No architecture policies found [ 4.286882] evm: Initialising EVM extended attributes: [ 4.289262] evm: security.selinux [ 4.290682] evm: security.ima [ 4.292364] evm: security.capability [ 4.294449] evm: HMAC attrs: 0x1 [ 4.298247] rtc_cmos 00:05: setting system clock to 2026-03-15 20:36:10 UTC (1773606970) [ 4.306676] debug: unmapping init [mem 0xffffffffa3803000-0xffffffffa39fffff] [ 4.310385] debug: unmapping init [mem 0xffffffffa2582000-0xffffffffa2858fff] [ 4.322645] Write protecting the kernel read-only data: 28672k [ 4.328847] debug: unmapping init [mem 0xffffffffa0c03000-0xffffffffa0dfffff] [ 4.332613] debug: unmapping init [mem 0xffffffffa1514000-0xffffffffa15fffff] [ 4.379841] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.390265] systemd[1]: Detected virtualization kvm. [ 4.392930] systemd[1]: Detected architecture x86-64. [ 4.394790] systemd[1]: Running in initial RAM disk. [ 4.423536] systemd[1]: No hostname configured. [ 4.425694] systemd[1]: Set hostname to . [ 4.428279] random: systemd: uninitialized urandom read (16 bytes read) [ 4.431652] systemd[1]: Initializing machine ID from random generator. [ 4.501107] random: ln: uninitialized urandom read (6 bytes read) [ 4.624221] random: systemd: uninitialized urandom read (16 bytes read) [ 4.628704] systemd[1]: Reached target Local File Systems. [ 4.634822] systemd[1]: Reached target Timers. [ 4.642348] systemd[1]: Listening on Journal Socket. [ 5.523462] device-mapper: uevent: version 1.0.3 [ 5.526721] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 6.355829] virtio_net virtio0 ens2: renamed from eth0 [ 6.390481] random: fast init done [ 6.391271] libata version 3.00 loaded. [ 6.405774] ata_piix 0000:00:01.1: version 2.13 [ 6.429181] scsi host0: ata_piix [ 6.462716] scsi host1: ata_piix [ 6.464480] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.466594] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.838944] random: crng init done [ 10.845673] random: 7 urandom warning(s) missed due to ratelimiting [ 14.067499] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 18.055962] printk: systemd: 25 output lines suppressed due to ratelimiting [ 19.245130] SELinux: Disabled at runtime. [ 19.466608] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.499480] systemd[1]: Detected virtualization kvm. [ 19.508480] systemd[1]: Detected architecture x86-64. [ 21.639714] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.665083] systemd[1]: Stopped Switch Root. [ 21.691872] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.703844] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.730189] systemd[1]: Stopped Journal Service. [ 21.777263] systemd[1]: Starting Journal Service... [ 21.790130] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ 22.082272] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 23.668204] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 25.144220] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.352442] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.459426] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.611528] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 26.611554] EDAC sbridge: Ver: 1.1.2 [ 32.655925] Key type dns_resolver registered [ 33.752939] NFS: Registering the id_resolver key type [ 33.763447] Key type id_resolver registered [ 33.766904] Key type id_legacy registered [ 69.593606] hrtimer: interrupt took 3162932 ns [ 125.133280] libcfs: loading out-of-tree module taints kernel. [ 125.322285] Key type ._llcrypt registered [ 125.336862] Key type .llcrypt registered [ 125.707148] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 153.711509] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 156.312754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 156.371310] alg: No test for adler32 (adler32-zlib) [ 158.686883] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 160.761963] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 162.784793] Key type lgssc registered [ 165.784666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 192.065295] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 259.805294] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 280.984360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 281.087338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 282.855443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 282.991841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 283.279588] Lustre: lustre-MDT0000: new disk, initializing [ 283.510708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 283.595487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 292.435696] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 322.112974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 322.272933] Lustre: 6533:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 322.311485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 322.315665] Lustre: Skipped 1 previous similar message [ 322.519231] Lustre: lustre-MDT0001: new disk, initializing [ 322.615586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 322.658145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 322.669762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 329.248374] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 335.976382] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 350.153937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 350.431719] Lustre: lustre-OST0000: new disk, initializing [ 350.434282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 350.488183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 358.127887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 358.166595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 358.367902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 361.015744] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 387.212840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 387.620370] Lustre: lustre-OST0001: new disk, initializing [ 387.636507] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 387.784674] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 394.832364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 394.850276] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 395.020610] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 398.064926] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 415.040890] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 423.251533] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 438.212943] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing check_logdir /tmp/testlogs/ [ 445.157779] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing yml_node [ 452.685779] Lustre: DEBUG MARKER: Client: 2.17.51.2 [ 456.807081] Lustre: DEBUG MARKER: MDS: 2.17.51.2 [ 460.483689] Lustre: DEBUG MARKER: OSS: 2.17.51.2 [ 462.953865] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Sun Mar 15 16:43:46 EDT 2026 [ 492.017683] Lustre: DEBUG MARKER: excepting tests: 18b 23b [ 508.092263] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 522.209437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 522.228350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.366904] Lustre: server umount lustre-MDT0000 complete [ 523.247092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.270270] Lustre: Skipped 1 previous similar message [ 528.356371] LustreError: 6544:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.394878] LustreError: 6544:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 9 previous similar messages [ 533.475768] LustreError: 6539:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.504668] LustreError: 6539:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 535.107404] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773607501 with bad export cookie 17409217355121147229 [ 535.111669] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.135085] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 536.202958] Lustre: server umount lustre-MDT0001 complete [ 554.783136] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773607504/real 1773607504] req@ffffa0c138ae2d80 x1859761865883520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773607520 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 554.805392] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.832684] Lustre: Skipped 1 previous similar message [ 555.999756] Lustre: 3658:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773607506/real 1773607506] req@ffffa0c13f54e300 x1859761865883776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773607522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 558.949368] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773607509/real 1773607509] req@ffffa0c13f5cdf80 x1859761865884032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773607525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 560.252197] Lustre: server umount lustre-OST0000 complete [ 561.119246] Lustre: 3657:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773607511/real 1773607511] req@ffffa0c004b0d500 x1859761865884416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773607527 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 567.267595] Lustre: 3658:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773607517/real 1773607517] req@ffffa0c138ae3480 x1859761865884672/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773607533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 571.960864] Lustre: server umount lustre-OST0001 complete [ 596.327066] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 601.651940] Key type lgssc unregistered [ 602.289144] LNet: 14778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.303340] LNetError: 14778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.352558] LNet: Removed LNI 192.168.201.114@tcp [ 604.354423] Key type .llcrypt unregistered [ 604.356076] Key type ._llcrypt unregistered [ 647.941831] Key type ._llcrypt registered [ 647.952740] Key type .llcrypt registered [ 648.188138] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 670.182152] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 672.216415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.271043] alg: No test for adler32 (adler32-zlib) [ 673.751465] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 674.012333] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 675.703294] Key type lgssc registered [ 677.929672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 757.867827] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 777.352221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 777.424458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.950332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 779.010367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 779.166217] Lustre: lustre-MDT0000: new disk, initializing [ 779.275642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.311904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 786.332453] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 808.277206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.502990] Lustre: 19214:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 808.597562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 808.603589] Lustre: Skipped 1 previous similar message [ 808.704252] Lustre: lustre-MDT0001: new disk, initializing [ 808.817199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 808.870216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 808.886456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 815.611296] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 822.900629] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 839.085169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 839.596290] Lustre: lustre-OST0000: new disk, initializing [ 839.624982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 839.808338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.423536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 848.456588] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 848.573433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 851.649169] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 874.661332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 874.877063] Lustre: lustre-OST0001: new disk, initializing [ 874.883903] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 874.959741] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 884.270279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 884.287749] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 884.338327] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 884.431472] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 902.380140] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 915.888790] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 933.847497] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 16:51:37 (1773607897) === [ 938.346957] Lustre: DEBUG MARKER: == sanity-lfsck test 0: Control LFSCK manually =========== 16:51:42 (1773607902) [ 938.761586] Lustre: 19222:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 938.775765] Lustre: 19222:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 938.785613] Lustre: 19222:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 938.813038] Lustre: 19222:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 938.833649] Lustre: 19222:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 938.853633] Lustre: 19222:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 939.262738] Lustre: 22609:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 939.271575] Lustre: 22609:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 939.281464] Lustre: 22609:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 939.288758] Lustre: 22609:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 939.296490] Lustre: 22609:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 939.306239] Lustre: 22609:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 939.317376] Lustre: 22609:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/12/0, punch: 0/0/0, quota 1/3/0 [ 939.324543] Lustre: 22609:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 939.341193] Lustre: 22609:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 939.353788] Lustre: 22609:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 939.371160] Lustre: 22609:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 939.388977] Lustre: 22609:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 940.338755] Lustre: 19222:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 940.348628] Lustre: 19222:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 20 previous similar messages [ 940.355605] Lustre: 19222:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 940.363443] Lustre: 19222:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 20 previous similar messages [ 940.376853] Lustre: 19222:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 940.388511] Lustre: 19222:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 20 previous similar messages [ 940.399448] Lustre: 19222:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/12/0, punch: 0/0/0, quota 1/3/0 [ 940.406471] Lustre: 19222:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 20 previous similar messages [ 940.415417] Lustre: 19222:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 940.422492] Lustre: 19222:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 20 previous similar messages [ 940.433527] Lustre: 19222:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 940.447374] Lustre: 19222:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 20 previous similar messages [ 942.380046] Lustre: 19222:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 942.391719] Lustre: 19222:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 83 previous similar messages [ 942.395234] Lustre: 19222:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 942.400077] Lustre: 19222:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 83 previous similar messages [ 942.403561] Lustre: 19222:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 942.410203] Lustre: 19222:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 83 previous similar messages [ 942.417397] Lustre: 19222:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/12/0, punch: 0/0/0, quota 1/3/0 [ 942.424275] Lustre: 19222:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 83 previous similar messages [ 942.430895] Lustre: 19222:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 942.437901] Lustre: 19222:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 83 previous similar messages [ 942.449840] Lustre: 19222:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 942.452480] Lustre: 19222:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 83 previous similar messages [ 946.399576] Lustre: 19222:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 946.416898] Lustre: 19222:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 119 previous similar messages [ 946.429818] Lustre: 19222:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 946.448492] Lustre: 19222:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 119 previous similar messages [ 946.462774] Lustre: 19222:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 946.478357] Lustre: 19222:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 119 previous similar messages [ 946.491994] Lustre: 19222:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/12/0, punch: 0/0/0, quota 1/3/0 [ 946.505512] Lustre: 19222:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 119 previous similar messages [ 946.518479] Lustre: 19222:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 946.527493] Lustre: 19222:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 119 previous similar messages [ 946.541671] Lustre: 19222:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 946.555194] Lustre: 19222:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 119 previous similar messages [ 950.321614] Lustre: *** cfs_fail_loc=1600, val=3*** [ 953.383275] Lustre: *** cfs_fail_loc=1600, val=3*** [ 958.231817] Lustre: *** cfs_fail_loc=1600, val=3*** [ 972.355395] Lustre: 23385:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0001: opcode 2: before 259 < left 274, rollback = 2 [ 972.366128] Lustre: 23310:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 972.375983] Lustre: 23385:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 99 previous similar messages [ 972.376779] Lustre: 23385:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 972.376784] Lustre: 23385:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 98 previous similar messages [ 972.376791] Lustre: 23385:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 972.376794] Lustre: 23385:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 98 previous similar messages [ 972.376800] Lustre: 23385:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 972.376804] Lustre: 23385:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 98 previous similar messages [ 972.376810] Lustre: 23385:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 972.376814] Lustre: 23385:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 98 previous similar messages [ 972.549163] Lustre: 23310:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 105 previous similar messages [ 976.357135] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.365353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 976.373935] Lustre: Skipped 3 previous similar messages [ 976.392995] Lustre: Skipped 3 previous similar messages [ 981.485930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 981.511561] Lustre: Skipped 3 previous similar messages [ 986.592703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 986.604884] Lustre: Skipped 3 previous similar messages [ 990.175509] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 990.530893] Lustre: server umount lustre-MDT0000 complete [ 996.583632] LustreError: 19208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773607962 with bad export cookie 9616351193041629620 [ 996.595274] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 996.608572] LustreError: 19208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 996.834375] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.838174] LustreError: 19222:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.851454] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 996.860583] Lustre: Skipped 1 previous similar message [ 996.882965] LustreError: 19222:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 997.220683] Lustre: server umount lustre-MDT0001 complete [ 1004.164562] Lustre: server umount lustre-OST0000 complete [ 1010.080471] Lustre: server umount lustre-OST0001 complete [ 1022.405521] Lustre: DEBUG MARKER: == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 16:53:06 (1773607986) [ 1044.338241] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1061.482337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1062.139690] LustreError: 26117:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1062.246663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.489888] LustreError: 26118:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1070.039428] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1072.608829] LustreError: 26118:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.744684] LustreError: 26117:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.415842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.952942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1092.780336] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1097.800344] Lustre: 27227:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1109.373418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1109.966578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.030991] LustreError: 27582:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1111.082367] LustreError: 27582:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1118.763719] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1131.524531] LustreError: 27582:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1131.574817] LustreError: 27582:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1132.248415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1132.556519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 1133.985718] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:65) [ 1142.689442] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1154.633626] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1162.028616] Lustre: 29074:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1169.490249] Lustre: 26112:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1169.499574] Lustre: 26112:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 19 previous similar messages [ 1169.506487] Lustre: 26112:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1169.512378] Lustre: 26112:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 13 previous similar messages [ 1169.516933] Lustre: 26112:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1169.520539] Lustre: 26112:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 20 previous similar messages [ 1169.525735] Lustre: 26112:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1169.533169] Lustre: 26112:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 20 previous similar messages [ 1169.538938] Lustre: 26112:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1169.544528] Lustre: 26112:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 20 previous similar messages [ 1169.552881] Lustre: 26112:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1169.562679] Lustre: 26112:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 20 previous similar messages [ 1179.081618] Lustre: *** cfs_fail_loc=1501, val=0*** [ 1193.132784] Lustre: Failing over lustre-MDT0000 [ 1193.445181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1193.460949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.486730] LustreError: 26835:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.720725] Lustre: server umount lustre-MDT0000 complete [ 1210.314945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1210.569916] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1211.019381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.032411] Lustre: Skipped 1 previous similar message [ 1211.125730] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1216.489319] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1216.489944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1216.574682] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1216.645782] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 1216.658806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:106 to 0x280000401:129) [ 1218.547685] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1223.615863] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1234.437786] Lustre: DEBUG MARKER: == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 16:56:38 (1773608198) [ 1236.694686] Lustre: 26111:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 1236.713298] Lustre: 26111:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 326 previous similar messages [ 1236.730283] Lustre: 26111:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 1236.746097] Lustre: 26111:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 326 previous similar messages [ 1236.766459] Lustre: 26111:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1236.784320] Lustre: 26111:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 326 previous similar messages [ 1236.798386] Lustre: 26111:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1236.822455] Lustre: 26111:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 326 previous similar messages [ 1236.837676] Lustre: 26111:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 1236.851789] Lustre: 26111:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 326 previous similar messages [ 1236.879809] Lustre: 26111:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1236.895961] Lustre: 26111:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 326 previous similar messages [ 1247.097941] Lustre: *** cfs_fail_loc=1502, val=0*** [ 1261.003812] Lustre: Failing over lustre-MDT0000 [ 1261.511773] Lustre: server umount lustre-MDT0000 complete [ 1262.563833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.568101] LustreError: 26835:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.589854] Lustre: Skipped 5 previous similar messages [ 1262.638926] LustreError: 26835:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 19 previous similar messages [ 1277.724488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1277.877679] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1278.365381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.470449] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1283.569349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1283.569721] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1283.590506] Lustre: Skipped 3 previous similar messages [ 1283.646294] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1283.711578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 1283.711728] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:170 to 0x2c0000401:193) [ 1285.341946] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1289.508411] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1302.056738] Lustre: DEBUG MARKER: == sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 16:57:45 (1773608265) [ 1304.308408] Lustre: 26113:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 1304.316041] Lustre: 26113:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 326 previous similar messages [ 1304.325071] Lustre: 26113:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 1304.334919] Lustre: 26113:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 326 previous similar messages [ 1304.347270] Lustre: 26113:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1304.356097] Lustre: 26113:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 326 previous similar messages [ 1304.363904] Lustre: 26113:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1304.372475] Lustre: 26113:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 326 previous similar messages [ 1304.387404] Lustre: 26113:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 1304.399410] Lustre: 26113:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 326 previous similar messages [ 1304.411911] Lustre: 26113:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1304.420480] Lustre: 26113:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 326 previous similar messages [ 1318.222169] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1318.240969] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1318.257370] Lustre: Skipped 1 previous similar message [ 1333.805826] Lustre: Failing over lustre-MDT0000 [ 1334.755315] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.771359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1334.811346] Lustre: Skipped 4 previous similar messages [ 1335.067501] Lustre: server umount lustre-MDT0000 complete [ 1350.855395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1351.136019] Lustre: 16367:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608300/real 1773608300] req@ffffa0c13f54d880 x1859762405108736/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773608316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1351.198377] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1360.932592] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x85742475e0a74034 [ 1360.980462] Lustre: MGC192.168.201.114@tcp: Connection restored to 0@lo (at 0@lo) [ 1360.999993] Lustre: Skipped 3 previous similar messages [ 1361.552025] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1366.500112] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1366.512531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1366.573676] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1366.711749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:234 to 0x280000401:257) [ 1366.712088] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:257) [ 1369.438059] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1375.734448] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1388.487804] Lustre: DEBUG MARKER: == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 16:59:12 (1773608352) [ 1403.222819] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1416.692022] Lustre: Failing over lustre-MDT0000 [ 1417.564402] Lustre: server umount lustre-MDT0000 complete [ 1417.696516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1417.706050] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.721432] LustreError: 26118:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1417.737852] Lustre: Skipped 2 previous similar messages [ 1417.783438] LustreError: 26118:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 43 previous similar messages [ 1433.425099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1433.874473] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1434.514438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.537876] Lustre: Skipped 1 previous similar message [ 1434.658419] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1439.717703] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1439.766880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1439.781677] Lustre: Skipped 3 previous similar messages [ 1439.828084] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1439.940599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:297 to 0x280000401:321) [ 1439.942817] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:298 to 0x2c0000401:321) [ 1442.927601] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1460.871173] Lustre: DEBUG MARKER: == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 17:00:23 (1773608423) [ 1463.414879] Lustre: 28354:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 1463.429498] Lustre: 28354:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 652 previous similar messages [ 1463.435793] Lustre: 28354:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 1463.443112] Lustre: 28354:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 652 previous similar messages [ 1463.465118] Lustre: 28354:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1463.472651] Lustre: 28354:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 652 previous similar messages [ 1463.480521] Lustre: 28354:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1463.492988] Lustre: 28354:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 652 previous similar messages [ 1463.515834] Lustre: 28354:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 1463.521482] Lustre: 28354:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 652 previous similar messages [ 1463.536085] Lustre: 28354:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1463.544985] Lustre: 28354:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 652 previous similar messages [ 1473.589158] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1485.496416] Lustre: Failing over lustre-MDT0000 [ 1485.816633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1485.831920] Lustre: Skipped 2 previous similar messages [ 1485.848852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1485.985469] Lustre: server umount lustre-MDT0000 complete [ 1503.780514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1504.190329] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1504.826584] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1509.864609] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1509.881528] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1509.924182] Lustre: Skipped 3 previous similar messages [ 1509.954523] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1510.067965] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:362 to 0x2c0000401:385) [ 1510.073912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:361 to 0x280000401:385) [ 1511.041253] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1526.460608] Lustre: DEBUG MARKER: == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 17:01:29 (1773608489) [ 1540.202793] Lustre: *** cfs_fail_loc=1605, val=0*** [ 1553.185256] Lustre: Failing over lustre-MDT0000 [ 1553.662709] Lustre: server umount lustre-MDT0000 complete [ 1555.951305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.990177] Lustre: Skipped 3 previous similar messages [ 1556.000625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1571.913824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1572.257462] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1572.862724] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1577.959771] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1577.987930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1578.008843] Lustre: Skipped 3 previous similar messages [ 1578.057947] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1578.162727] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:425 to 0x2c0000401:449) [ 1578.170164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:426 to 0x280000401:449) [ 1580.839879] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1593.677664] Lustre: DEBUG MARKER: == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 17:02:37 (1773608557) [ 1603.778928] Lustre: *** cfs_fail_loc=161d, val=0*** [ 1619.605108] Lustre: Failing over lustre-MDT0000 [ 1620.024098] Lustre: server umount lustre-MDT0000 complete [ 1624.042747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1624.077766] Lustre: Skipped 4 previous similar messages [ 1636.392332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1636.620174] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1637.150802] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1642.467782] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1642.491080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1642.493838] Lustre: Skipped 3 previous similar messages [ 1642.546198] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1642.597419] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:490 to 0x2c0000401:513) [ 1642.597720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:489 to 0x280000401:513) [ 1644.538826] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1659.188174] Lustre: DEBUG MARKER: == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 17:03:42 (1773608622) [ 1663.740668] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1683.172923] Lustre: DEBUG MARKER: == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 17:04:06 (1773608646) [ 1693.942565] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1695.780879] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1714.457638] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 17:04:38 (1773608678) [ 1719.461383] Lustre: 32390:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 1719.474704] Lustre: 32390:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1429 previous similar messages [ 1719.488771] Lustre: 32390:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1719.499526] Lustre: 32390:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1429 previous similar messages [ 1719.508050] Lustre: 32390:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1719.521260] Lustre: 32390:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1429 previous similar messages [ 1719.531406] Lustre: 32390:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/12/0, punch: 0/0/0, quota 1/3/0 [ 1719.539688] Lustre: 32390:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1429 previous similar messages [ 1719.545056] Lustre: 32390:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1719.556430] Lustre: 32390:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1429 previous similar messages [ 1719.573359] Lustre: 32390:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1719.582140] Lustre: 32390:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1429 previous similar messages [ 1750.986580] Lustre: Failing over lustre-MDT0000 [ 1751.443777] Lustre: server umount lustre-MDT0000 complete [ 1755.110151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.120059] LustreError: 28354:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1755.126477] Lustre: Skipped 4 previous similar messages [ 1755.163684] LustreError: 28354:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 59 previous similar messages [ 1759.374385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1771.487845] Lustre: 16366:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608721/real 1773608721] req@ffffa0c00b219880 x1859762405558656/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773608737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1771.526369] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1774.965914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1791.950397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1791.997425] Lustre: lustre-MDT0000: reset Object Index mappings [ 1797.095652] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x85742475e0a889a4 [ 1797.106090] Lustre: MGC192.168.201.114@tcp: Connection restored to 0@lo (at 0@lo) [ 1797.116158] Lustre: Skipped 3 previous similar messages [ 1797.429575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.450274] Lustre: Skipped 3 previous similar messages [ 1797.491517] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1802.735609] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1802.803129] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1802.890236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:593 to 0x280000401:609) [ 1802.893352] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:593 to 0x2c0000401:609) [ 1803.953918] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1809.894060] LustreError: 42809:0:(lfsck_engine.c:1018:lfsck_master_engine()) lustre-MDT0000-osd: master engine fail to verify the .lustre/lost+found/, go ahead: rc = -115 [ 1809.919558] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1810.976029] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1812.005891] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1814.049655] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1814.054830] Lustre: Skipped 1 previous similar message [ 1824.834062] Lustre: Failing over lustre-MDT0000 [ 1825.760636] Lustre: server umount lustre-MDT0000 complete [ 1828.322826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1844.703373] Lustre: 16364:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608794/real 1773608794] req@ffffa0c12f59f480 x1859762405597952/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773608810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1845.547272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1853.930930] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x85742475e0a88cad [ 1860.227163] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:593 to 0x2c0000401:641) [ 1860.237808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:593 to 0x280000401:641) [ 1863.763844] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1868.228800] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1880.025522] Lustre: DEBUG MARKER: == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 17:07:24 (1773608844) [ 1883.523783] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1901.494081] Lustre: Failing over lustre-MDT0000 [ 1902.069742] Lustre: server umount lustre-MDT0000 complete [ 1913.401159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1922.530991] Lustre: 16367:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608872/real 1773608872] req@ffffa0c12f339c00 x1859762405680256/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773608888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1922.648531] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1922.666037] LustreError: Skipped 1 previous similar message [ 1931.873617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1950.969525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1951.008233] Lustre: lustre-MDT0000: reset Object Index mappings [ 1967.071741] LustreError: 16363:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0c12dbd8a80 x1859762405697920/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1967.791837] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1967.809491] Lustre: Skipped 1 previous similar message [ 1972.712580] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1972.743923] Lustre: Skipped 1 previous similar message [ 1972.761654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1972.782794] Lustre: Skipped 9 previous similar messages [ 1972.805452] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1972.812310] Lustre: Skipped 1 previous similar message [ 1972.883854] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:682 to 0x2c0000401:705) [ 1972.890118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:705) [ 1974.524418] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1980.217741] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1980.226203] Lustre: Skipped 2 previous similar messages [ 1988.448233] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1988.462735] Lustre: Skipped 7 previous similar messages [ 2000.708040] Lustre: Failing over lustre-MDT0000 [ 2001.884494] Lustre: server umount lustre-MDT0000 complete [ 2003.438050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2019.807261] Lustre: 16364:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608969/real 1773608969] req@ffffa0c12f33a300 x1859762405724160/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773608985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2019.928284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2035.867762] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:682 to 0x2c0000401:737) [ 2035.880192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:737) [ 2037.982039] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2043.135819] Lustre: *** cfs_fail_loc=1505, val=0*** [ 2043.148284] Lustre: Skipped 85 previous similar messages [ 2055.512757] Lustre: DEBUG MARKER: == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 17:10:19 (1773609019) [ 2070.716204] Lustre: *** cfs_fail_loc=1600, val=1*** [ 2070.727635] Lustre: Skipped 3 previous similar messages [ 2098.200383] Lustre: DEBUG MARKER: == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 17:11:02 (1773609062) [ 2112.201621] Lustre: *** cfs_fail_loc=1601, val=1*** [ 2112.206344] Lustre: Skipped 11 previous similar messages [ 2148.315283] Lustre: DEBUG MARKER: == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 17:11:52 (1773609112) [ 2175.625880] Lustre: Failing over lustre-MDT0000 [ 2176.237384] Lustre: server umount lustre-MDT0000 complete [ 2179.039519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2179.055442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.079700] Lustre: Skipped 15 previous similar messages [ 2190.599521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2190.776744] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2190.805775] LustreError: Skipped 1 previous similar message [ 2191.362699] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2191.389607] Lustre: Skipped 1 previous similar message [ 2196.457027] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2196.480340] Lustre: Skipped 1 previous similar message [ 2196.509046] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2196.533917] Lustre: Skipped 1 previous similar message [ 2196.623706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:855 to 0x280000401:897) [ 2196.623757] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:855 to 0x2c0000401:897) [ 2198.414935] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2212.476774] Lustre: DEBUG MARKER: == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 17:12:56 (1773609176) [ 2236.131583] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2269.221757] Lustre: 52864:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2308.403568] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2313.955054] Lustre: 54002:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2322.590954] Lustre: 26112:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 2322.604560] Lustre: 26112:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1659 previous similar messages [ 2322.619082] Lustre: 26112:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 2322.628905] Lustre: 26112:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1659 previous similar messages [ 2322.641135] Lustre: 26112:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 2322.650948] Lustre: 26112:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1659 previous similar messages [ 2322.658160] Lustre: 26112:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 2322.675616] Lustre: 26112:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1659 previous similar messages [ 2322.694608] Lustre: 26112:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 2322.706168] Lustre: 26112:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1659 previous similar messages [ 2322.732302] Lustre: 26112:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 2322.750700] Lustre: 26112:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1659 previous similar messages [ 2336.150627] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2336.153255] Lustre: Skipped 82 previous similar messages [ 2340.818327] Lustre: *** cfs_fail_loc=1602, val=1*** [ 2341.856979] Lustre: *** cfs_fail_loc=1602, val=1*** [ 2342.946030] Lustre: *** cfs_fail_loc=1602, val=1*** [ 2344.992621] Lustre: *** cfs_fail_loc=1602, val=1*** [ 2344.995128] Lustre: Skipped 1 previous similar message [ 2349.087742] Lustre: *** cfs_fail_loc=1602, val=1*** [ 2349.102451] Lustre: Skipped 3 previous similar messages [ 2350.184261] Lustre: Failing over lustre-MDT0000 [ 2350.769134] Lustre: server umount lustre-MDT0000 complete [ 2355.169601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2355.172937] LustreError: 26835:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.224552] LustreError: 26835:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 167 previous similar messages [ 2366.822649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2367.614872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.631339] Lustre: Skipped 4 previous similar messages [ 2372.599669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2372.623624] Lustre: Skipped 11 previous similar messages [ 2372.788245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:947 to 0x280000401:993) [ 2372.788930] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:947 to 0x2c0000401:993) [ 2375.246562] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2390.327343] Lustre: DEBUG MARKER: == sanity-lfsck test 8: LFSCK state machine ============== 17:15:54 (1773609354) [ 2398.187985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2398.209571] Lustre: Skipped 4 previous similar messages [ 2400.416264] Lustre: server umount lustre-MDT0000 complete [ 2405.623385] LustreError: 29076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773609371 with bad export cookie 9616351193041843666 [ 2405.645493] LustreError: 29076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2406.238662] Lustre: server umount lustre-MDT0001 complete [ 2423.492418] Lustre: server umount lustre-OST0000 complete [ 2424.287219] Lustre: 16366:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773609374/real 1773609374] req@ffffa0c13f710000 x1859762406084736/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773609390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2430.120096] Lustre: server umount lustre-OST0001 complete [ 2441.468756] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 2454.404446] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2539.806554] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2560.727918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2561.245583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2561.324316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2561.494856] Lustre: lustre-MDT0000: new disk, initializing [ 2561.727920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2569.590948] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2587.759013] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.023551] Lustre: 59041:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2588.106476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2588.113628] Lustre: Skipped 1 previous similar message [ 2588.294267] Lustre: lustre-MDT0001: new disk, initializing [ 2588.548067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2588.577665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2595.638658] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2602.639934] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2614.253747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2614.778331] Lustre: lustre-OST0000: new disk, initializing [ 2614.784517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2616.530471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2616.548326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2616.659072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2626.029791] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2642.326448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2642.622606] Lustre: lustre-OST0001: new disk, initializing [ 2642.641239] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2644.252252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2644.269464] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2644.415678] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2654.842447] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2670.473950] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2677.894111] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2701.864368] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2703.553076] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2703.568143] Lustre: Skipped 19 previous similar messages [ 2711.724734] Lustre: *** cfs_fail_loc=1601, val=2*** [ 2711.728774] Lustre: Skipped 24 previous similar messages [ 2735.988527] Lustre: Failing over lustre-MDT0000 [ 2736.502616] Lustre: server umount lustre-MDT0000 complete [ 2736.624142] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.652082] Lustre: Skipped 10 previous similar messages [ 2750.394548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2750.543980] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2750.552746] LustreError: Skipped 2 previous similar messages [ 2750.854326] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2750.896502] Lustre: Skipped 1 previous similar message [ 2756.068222] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2756.096033] Lustre: Skipped 1 previous similar message [ 2756.139151] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2756.151872] Lustre: Skipped 1 previous similar message [ 2756.205713] Lustre: *** cfs_fail_loc=160b, val=2*** [ 2756.210695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 2756.212102] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:65) [ 2757.461773] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2769.808494] Lustre: Failing over lustre-MDT0000 [ 2770.137595] Lustre: server umount lustre-MDT0000 complete [ 2771.423709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2771.445037] LustreError: Skipped 1 previous similar message [ 2783.264869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2788.963145] Lustre: *** cfs_fail_loc=160b, val=2*** [ 2788.982030] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:97) [ 2788.982060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 2791.417514] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2800.978311] Lustre: Failing over lustre-MDT0000 [ 2801.359162] Lustre: server umount lustre-MDT0000 complete [ 2815.340408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2821.186287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:129) [ 2821.186582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 2822.747400] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2831.400921] Lustre: *** cfs_fail_loc=1602, val=2*** [ 2831.404955] Lustre: Skipped 1 previous similar message [ 2847.861888] Lustre: DEBUG MARKER: == sanity-lfsck test 9a: LFSCK speed control (1) ========= 17:23:31 (1773609811) [ 2867.417955] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2894.221719] Lustre: 68373:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2930.918403] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2937.382372] Lustre: 69511:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2961.393167] Lustre: 59047:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 2961.401916] Lustre: 59047:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1671 previous similar messages [ 2961.419023] Lustre: 59047:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 2961.429387] Lustre: 59047:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1671 previous similar messages [ 2961.437773] Lustre: 59047:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 2961.448395] Lustre: 59047:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1671 previous similar messages [ 2961.465805] Lustre: 59047:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 2961.475511] Lustre: 59047:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1671 previous similar messages [ 2961.490821] Lustre: 59047:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 2961.509733] Lustre: 59047:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1671 previous similar messages [ 2961.521491] Lustre: 59047:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 2961.531451] Lustre: 59047:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1671 previous similar messages [ 3117.514561] Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) ========= 17:28:01 (1773610081) [ 3194.601279] Lustre: *** cfs_fail_loc=1604, val=0*** [ 3194.613979] Lustre: Skipped 4 previous similar messages [ 3231.673916] Lustre: *** cfs_fail_loc=160c, val=0*** [ 3231.676191] Lustre: Skipped 11 previous similar messages [ 3275.626538] Lustre: DEBUG MARKER: == sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 17:30:40 (1773610240) [ 3343.782392] Lustre: *** cfs_fail_loc=1603, val=0*** [ 3344.293766] Lustre: *** cfs_fail_loc=1603, val=0*** [ 3344.304419] Lustre: Skipped 13 previous similar messages [ 3345.340852] Lustre: *** cfs_fail_loc=1603, val=0*** [ 3345.343176] Lustre: Skipped 20 previous similar messages [ 3347.345785] Lustre: *** cfs_fail_loc=1603, val=0*** [ 3347.352182] Lustre: Skipped 55 previous similar messages [ 3351.379962] Lustre: *** cfs_fail_loc=1603, val=0*** [ 3351.383391] Lustre: Skipped 141 previous similar messages [ 3359.387531] Lustre: *** cfs_fail_loc=1603, val=0*** [ 3359.392466] Lustre: Skipped 320 previous similar messages [ 3375.409777] Lustre: *** cfs_fail_loc=1603, val=0*** [ 3375.424041] Lustre: Skipped 635 previous similar messages [ 3407.419936] Lustre: *** cfs_fail_loc=1603, val=0*** [ 3407.422079] Lustre: Skipped 1155 previous similar messages [ 3436.943636] Lustre: *** cfs_fail_loc=1604, val=0*** [ 3436.946249] Lustre: Skipped 2599 previous similar messages [ 3691.832854] Lustre: 59047:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 3691.845079] Lustre: 59047:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 36829 previous similar messages [ 3691.854578] Lustre: 59047:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3691.859846] Lustre: 59047:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 36829 previous similar messages [ 3691.872274] Lustre: 59047:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3691.879189] Lustre: 59047:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 36829 previous similar messages [ 3691.887834] Lustre: 59047:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/12/1, punch: 0/0/0, quota 1/3/2 [ 3691.897886] Lustre: 59047:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 36829 previous similar messages [ 3691.904482] Lustre: 59047:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 3691.912734] Lustre: 59047:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 36829 previous similar messages [ 3691.921360] Lustre: 59047:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3691.929045] Lustre: 59047:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 36829 previous similar messages [ 3757.888568] Lustre: DEBUG MARKER: == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 17:38:41 (1773610721) [ 3993.398163] Lustre: server umount lustre-MDT0000 complete [ 3993.573306] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3993.574227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3993.586318] Lustre: Skipped 10 previous similar messages [ 3993.587525] LustreError: 62668:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.600344] LustreError: Skipped 1 previous similar message [ 3993.634563] LustreError: 62668:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 51 previous similar messages [ 3998.949917] LustreError: 59034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610965 with bad export cookie 9616351193041863049 [ 3998.959307] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3998.981057] LustreError: 59034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3999.035981] LustreError: Skipped 2 previous similar messages [ 3999.569308] Lustre: server umount lustre-MDT0001 complete [ 4016.522623] Lustre: server umount lustre-OST0000 complete [ 4020.703332] Lustre: 16364:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773610970/real 1773610970] req@ffffa0c00c64ad80 x1859762410034944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773610986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4020.749696] Lustre: 16364:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4022.552980] Lustre: server umount lustre-OST0001 complete [ 4032.759390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4045.709107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4061.349418] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4066.527798] LustreError: 74681:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.114@tcp: failed processing log, type 4: rc = -110 [ 4092.322100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4092.333167] Lustre: Skipped 7 previous similar messages [ 4101.870444] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4107.491604] Lustre: 75247:0:(ofd_dev.c:560:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 4107.516881] Lustre: *** cfs_fail_loc=160e, val=3*** [ 4110.579890] Lustre: 75247:0:(ofd_dev.c:572:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 4122.981410] Lustre: DEBUG MARKER: == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 17:44:46 (1773611086) [ 4144.721070] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4161.228237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4162.157082] LustreError: 74706:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.194818] LustreError: 74706:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 4162.499918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3272 to 0x280000401:3297) [ 4168.703570] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4180.774285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.304044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4181.318785] Lustre: Skipped 1 previous similar message [ 4188.498760] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4192.696643] Lustre: 77819:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4214.471304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4216.434744] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3209 to 0x2c0000401:3233) [ 4224.716174] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4234.624907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4239.344781] Lustre: 79306:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4250.905972] Lustre: *** cfs_fail_loc=160d, val=0*** [ 4251.829046] Lustre: *** cfs_fail_loc=160d, val=0*** [ 4251.847181] Lustre: Skipped 3 previous similar messages [ 4261.587935] Lustre: Failing over lustre-OST0000 [ 4261.701941] Lustre: server umount lustre-OST0000 complete [ 4263.391521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4263.413590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.455796] Lustre: Skipped 4 previous similar messages [ 4277.861809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4278.156099] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4278.183727] Lustre: Skipped 2 previous similar messages [ 4279.983108] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4280.013723] Lustre: Skipped 2 previous similar messages [ 4280.056155] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4280.060038] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4280.077720] Lustre: Skipped 2 previous similar messages [ 4280.088611] Lustre: *** cfs_fail_loc=215, val=0*** [ 4280.092733] Lustre: Skipped 15 previous similar messages [ 4280.112308] Lustre: Skipped 2 previous similar messages [ 4285.409220] Lustre: *** cfs_fail_loc=215, val=0*** [ 4285.417676] Lustre: Skipped 1 previous similar message [ 4288.316903] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4290.529183] Lustre: *** cfs_fail_loc=215, val=0*** [ 4290.539926] Lustre: Skipped 1 previous similar message [ 4294.303417] Lustre: 80696:0:(ofd_dev.c:560:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 4294.387836] Lustre: 80696:0:(ofd_dev.c:572:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 4295.647844] Lustre: *** cfs_fail_loc=215, val=0*** [ 4295.653172] Lustre: Skipped 1 previous similar message [ 4298.801044] Lustre: Failing over lustre-OST0000 [ 4299.131991] Lustre: server umount lustre-OST0000 complete [ 4313.247516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4315.676942] Lustre: *** cfs_fail_loc=215, val=0*** [ 4321.845155] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4325.857473] Lustre: *** cfs_fail_loc=215, val=0*** [ 4325.867297] Lustre: Skipped 6 previous similar messages [ 4334.051239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4334.068223] Lustre: Skipped 3 previous similar messages [ 4337.011754] Lustre: server umount lustre-MDT0000 complete [ 4339.171567] LustreError: 76700:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4339.219417] LustreError: 76700:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 39 previous similar messages [ 4343.117109] LustreError: 74687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611309 with bad export cookie 9616351193043444489 [ 4343.121117] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4343.138812] LustreError: 74687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4343.511804] Lustre: server umount lustre-MDT0001 complete [ 4360.672351] Lustre: 16366:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773611310/real 1773611310] req@ffffa0c102e27480 x1859762410153088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773611326 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4360.722450] Lustre: server umount lustre-OST0000 complete [ 4364.784350] Lustre: 16365:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773611314/real 1773611314] req@ffffa0c00c2cf480 x1859762410153472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773611330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4366.656841] Lustre: server umount lustre-OST0001 complete [ 4381.591709] Lustre: DEBUG MARKER: == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 17:49:05 (1773611345) [ 4402.349373] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4418.283237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.961188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4418.966810] Lustre: Skipped 3 previous similar messages [ 4426.872526] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4442.564241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.543468] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4455.278279] Lustre: 85028:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4466.465358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4478.583829] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4490.748507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3362 to 0x280000401:3393) [ 4493.554781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4495.267873] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3209 to 0x2c0000401:3265) [ 4503.206475] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4513.790557] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4519.969780] Lustre: 86872:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4526.953168] Lustre: 83913:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 278, rollback = 2 [ 4526.972629] Lustre: 83913:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 400 previous similar messages [ 4526.987239] Lustre: 83913:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 4526.999942] Lustre: 83913:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 400 previous similar messages [ 4527.028109] Lustre: 83913:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4527.051918] Lustre: 83913:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 400 previous similar messages [ 4527.063848] Lustre: 83913:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4527.072761] Lustre: 83913:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 400 previous similar messages [ 4527.104251] Lustre: 83913:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 4527.111566] Lustre: 83913:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 400 previous similar messages [ 4527.127175] Lustre: 83913:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4527.142519] Lustre: 83913:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 400 previous similar messages [ 4577.969821] Lustre: DEBUG MARKER: == sanity-lfsck test 12b: auto detect Lustre device ====== 17:52:21 (1773611541) [ 4599.473422] Lustre: DEBUG MARKER: == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 17:52:43 (1773611563) [ 4601.801561] Lustre: *** cfs_fail_loc=160f, val=0*** [ 4618.133836] Lustre: DEBUG MARKER: == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 17:53:01 (1773611581) [ 4624.006445] Lustre: *** cfs_fail_loc=1610, val=0*** [ 4624.017977] Lustre: Skipped 3 previous similar messages [ 4688.355194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.361365] LustreError: 87541:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4688.403430] Lustre: Skipped 9 previous similar messages [ 4688.434278] LustreError: 87541:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 16 previous similar messages [ 4689.292310] Lustre: server umount lustre-MDT0000 complete [ 4695.795199] LustreError: 89580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611661 with bad export cookie 9616351193043452931 [ 4695.802144] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4695.807140] LustreError: 89580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4696.201616] Lustre: server umount lustre-MDT0001 complete [ 4713.034183] Lustre: server umount lustre-OST0000 complete [ 4714.783358] Lustre: 16367:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773611664/real 1773611664] req@ffffa0c0108b7800 x1859762410419712/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773611680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4714.840040] Lustre: 16367:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4719.194205] Lustre: server umount lustre-OST0001 complete [ 4742.221698] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4759.460249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4760.405057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4760.418952] Lustre: Skipped 3 previous similar messages [ 4767.662090] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4782.410647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.352411] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4794.466589] Lustre: 92725:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4803.937975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4805.179671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3554 to 0x280000401:3585) [ 4808.246714] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:161) [ 4814.169192] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4828.964221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4831.360665] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3361) [ 4831.422377] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:161) [ 4838.931679] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4849.901310] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4855.417354] Lustre: 94573:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4869.110736] Lustre: DEBUG MARKER: == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 17:57:13 (1773611833) [ 4877.761861] Lustre: *** cfs_fail_loc=1610, val=0*** [ 4877.771437] Lustre: Skipped 63 previous similar messages [ 4911.077417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4911.094213] LustreError: Skipped 3 previous similar messages [ 4911.105631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4913.835545] Lustre: server umount lustre-MDT0000 complete [ 4920.248434] LustreError: 91599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611886 with bad export cookie 9616351193043481351 [ 4920.257255] LustreError: 91599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4921.102848] Lustre: server umount lustre-MDT0001 complete [ 4937.263893] Lustre: server umount lustre-OST0000 complete [ 4938.019693] Lustre: 16364:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773611888/real 1773611888] req@ffffa0c00cb60000 x1859762410568064/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773611904 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4938.077824] Lustre: 16364:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4943.221569] Lustre: server umount lustre-OST0001 complete [ 4966.965402] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4981.635152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4989.605742] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5001.225854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5007.498160] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5011.839198] Lustre: 98560:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5020.088663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5026.848153] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:193) [ 5027.837745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3682 to 0x280000401:3713) [ 5029.738574] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5042.682454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5044.974766] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3393) [ 5044.987598] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:193) [ 5051.161175] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5061.980721] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5085.240535] Lustre: DEBUG MARKER: == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 18:00:48 (1773612048) [ 5091.019344] Lustre: *** cfs_fail_loc=1611, val=0*** [ 5091.029037] Lustre: Skipped 63 previous similar messages [ 5091.333923] Lustre: *** cfs_fail_loc=1611, val=0*** [ 5108.512671] Lustre: DEBUG MARKER: == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 18:01:12 (1773612072) [ 5112.287392] Lustre: *** cfs_fail_loc=1612, val=0*** [ 5112.418882] Lustre: *** cfs_fail_loc=1612, val=0*** [ 5112.429114] Lustre: Skipped 3 previous similar messages [ 5127.824627] Lustre: DEBUG MARKER: == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 18:01:31 (1773612091) [ 5130.286085] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_15c MDS newer than 2.7.55, LU-6475 [ 5133.198455] Lustre: DEBUG MARKER: == sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 18:01:37 (1773612097) [ 5133.611290] Lustre: 99937:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 5133.622247] Lustre: 99937:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1335 previous similar messages [ 5133.629305] Lustre: 99937:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 5133.636080] Lustre: 99937:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1335 previous similar messages [ 5133.643448] Lustre: 99937:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5133.649855] Lustre: 99937:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1335 previous similar messages [ 5133.656982] Lustre: 99937:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 5133.662803] Lustre: 99937:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1335 previous similar messages [ 5133.668422] Lustre: 99937:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 5133.674789] Lustre: 99937:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1335 previous similar messages [ 5133.678753] Lustre: 99937:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5133.684209] Lustre: 99937:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1335 previous similar messages [ 5140.552307] Lustre: *** cfs_fail_loc=1709, val=0*** [ 5140.556469] LustreError: 97462:0:(mdt_reint.c:2621:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240002340:0x6c:0x0]/f21 failed: rc = -5 [ 5221.736838] Lustre: server umount lustre-MDT0000 complete [ 5221.870854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5221.896786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.925403] Lustre: Skipped 8 previous similar messages [ 5233.173882] LustreError: 98562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612199 with bad export cookie 9616351193043496086 [ 5233.174641] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5233.187637] LustreError: 98562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5233.202555] LustreError: Skipped 1 previous similar message [ 5233.532869] Lustre: server umount lustre-MDT0001 complete [ 5253.088828] Lustre: 16367:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773612203/real 1773612203] req@ffffa0c00cfe4000 x1859762411224320/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773612219 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5253.132391] Lustre: 16367:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5256.381276] Lustre: server umount lustre-OST0000 complete [ 5267.936557] Lustre: server umount lustre-OST0001 complete [ 5289.229235] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5292.828896] Key type lgssc unregistered [ 5293.213091] LNet: 104248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5293.228824] LNetError: 104248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5293.243560] LNet: Removed LNI 192.168.201.114@tcp [ 5294.595366] Key type .llcrypt unregistered [ 5294.603168] Key type ._llcrypt unregistered [ 5325.422471] Key type ._llcrypt registered [ 5325.428988] Key type .llcrypt registered [ 5325.573683] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 5341.828410] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5343.523641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5343.582585] alg: No test for adler32 (adler32-zlib) [ 5344.738820] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 5345.036319] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 5346.855327] Key type lgssc registered [ 5347.925219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5410.451351] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5428.084581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5428.177922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5429.630714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5429.682605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5429.798828] Lustre: lustre-MDT0000: new disk, initializing [ 5429.898488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5429.943226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5436.155924] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5453.397918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.609807] Lustre: 108658:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5453.647657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5453.652233] Lustre: Skipped 1 previous similar message [ 5453.775291] Lustre: lustre-MDT0001: new disk, initializing [ 5453.910231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5453.976898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5453.995524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5459.667412] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5466.124504] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5478.401109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5478.626837] Lustre: lustre-OST0000: new disk, initializing [ 5478.629855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5478.684598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5487.450285] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5488.211967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5488.228120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5488.358795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5503.893593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5504.057198] Lustre: lustre-OST0001: new disk, initializing [ 5504.063523] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5504.135148] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5513.197703] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5513.288572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5513.317298] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5513.426874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5526.459622] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5533.037616] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5545.963425] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 18:08:30 (1773612510) === [ 5554.201653] Lustre: DEBUG MARKER: == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 18:08:38 (1773612518) [ 5554.461103] Lustre: 108664:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 5554.476561] Lustre: 108664:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 5554.497528] Lustre: 108664:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5554.518041] Lustre: 108664:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 5554.531100] Lustre: 108664:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 5554.552271] Lustre: 108664:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5554.988041] Lustre: 108664:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 5555.006532] Lustre: 108664:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 6 previous similar messages [ 5555.017253] Lustre: 108664:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 5555.027951] Lustre: 108664:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 5555.035741] Lustre: 108664:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5555.049743] Lustre: 108664:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 5555.059404] Lustre: 108664:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 5555.076543] Lustre: 108664:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 5555.095105] Lustre: 108664:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 5555.108542] Lustre: 108664:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 5555.126892] Lustre: 108664:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5555.145734] Lustre: 108664:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 5556.018603] Lustre: 112395:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 5556.030479] Lustre: 112395:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 110 previous similar messages [ 5556.039466] Lustre: 112395:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 5556.045905] Lustre: 112395:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 110 previous similar messages [ 5556.057685] Lustre: 112395:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5556.071257] Lustre: 112395:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 110 previous similar messages [ 5556.079251] Lustre: 112395:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/12/0, punch: 0/0/0, quota 7/369/0 [ 5556.095694] Lustre: 112395:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 110 previous similar messages [ 5556.101516] Lustre: 112395:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 5556.107995] Lustre: 112395:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 110 previous similar messages [ 5556.126909] Lustre: 108666:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 5556.131561] Lustre: 108666:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 113 previous similar messages [ 5558.284630] Lustre: *** cfs_fail_loc=1613, val=0*** [ 5570.494849] Lustre: DEBUG MARKER: == sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 18:08:54 (1773612534) [ 5571.944536] Lustre: 108666:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 278, rollback = 2 [ 5571.960095] Lustre: 108666:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 191 previous similar messages [ 5571.967394] Lustre: 108666:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 5571.978342] Lustre: 108666:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 191 previous similar messages [ 5571.983658] Lustre: 108666:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5571.990490] Lustre: 108666:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 191 previous similar messages [ 5571.995615] Lustre: 108666:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 5572.004300] Lustre: 108666:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 191 previous similar messages [ 5572.014031] Lustre: 108666:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 5572.021728] Lustre: 108666:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 191 previous similar messages [ 5572.029996] Lustre: 108666:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5572.036338] Lustre: 108666:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 188 previous similar messages [ 5573.526912] Lustre: *** cfs_fail_loc=1614, val=0*** [ 5574.161378] Lustre: *** cfs_fail_loc=1614, val=103*** [ 5574.170153] Lustre: Skipped 1 previous similar message [ 5579.188948] Lustre: 110558:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 275, rollback = 2 [ 5579.201526] Lustre: 110558:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 11 previous similar messages [ 5579.212089] Lustre: 110558:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 5579.227640] Lustre: 110558:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 5579.241931] Lustre: 110558:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 3/275/0 [ 5579.258954] Lustre: 110558:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 5579.266382] Lustre: 110558:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/12/0, punch: 1/4/0, quota 7/225/0 [ 5579.281572] Lustre: 110558:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 5579.286826] Lustre: 110558:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 5579.296731] Lustre: 110558:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 5579.304260] Lustre: 110558:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 5579.311166] Lustre: 110558:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 5588.645207] Lustre: DEBUG MARKER: == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 18:09:12 (1773612552) [ 5589.133380] Lustre: 108666:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 5589.146817] Lustre: 108666:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 5589.159025] Lustre: 108666:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 5589.168401] Lustre: 108666:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5589.180301] Lustre: 108666:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5589.198415] Lustre: 108666:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5589.208110] Lustre: 108666:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 5589.215975] Lustre: 108666:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5589.224118] Lustre: 108666:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 5589.229853] Lustre: 108666:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5589.236696] Lustre: 108666:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5589.244985] Lustre: 108666:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5591.933088] Lustre: *** cfs_fail_loc=1615, val=0*** [ 5591.943987] Lustre: Skipped 1 previous similar message [ 5593.184724] Lustre: *** cfs_fail_loc=1615, val=0*** [ 5593.189781] Lustre: Skipped 1 previous similar message [ 5599.181859] LustreError: 113883:0:(lfsck_layout.c:2103:lfsck_layout_add_comp()) five: Five five five five five five file Hello [0x2c0000401:0x2:0x0] and [0x2c0000401:0x2:0x0]d: rc = 0 [ 5615.998098] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_18b skipping excluded test 18b [ 5618.740487] Lustre: DEBUG MARKER: == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 18:09:42 (1773612582) [ 5619.497088] Lustre: 108665:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 5619.506441] Lustre: 108665:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 21 previous similar messages [ 5619.517290] Lustre: 108665:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 5619.523693] Lustre: 108665:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 5619.530520] Lustre: 108665:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5619.540356] Lustre: 108665:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 5619.545779] Lustre: 108665:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 5619.550464] Lustre: 108665:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 5619.558199] Lustre: 108665:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 5619.573599] Lustre: 108665:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 5619.580553] Lustre: 108665:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5619.587322] Lustre: 108665:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 5622.283427] Lustre: *** cfs_fail_loc=1617, val=0*** [ 5622.427320] Lustre: *** cfs_fail_loc=1617, val=0*** [ 5625.406868] Lustre: *** cfs_fail_loc=1616, val=0*** [ 5625.409241] Lustre: Skipped 3 previous similar messages [ 5647.177449] Lustre: DEBUG MARKER: == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 18:10:11 (1773612611) [ 5649.593737] Lustre: *** cfs_fail_loc=1618, val=0*** [ 5649.598825] Lustre: Skipped 5 previous similar messages [ 5684.192485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5684.200336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5684.219270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5687.265281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5687.271231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5687.301457] Lustre: Skipped 2 previous similar messages [ 5687.319641] Lustre: Skipped 3 previous similar messages [ 5689.998698] Lustre: server umount lustre-MDT0000 complete [ 5694.610456] LustreError: 109598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612660 with bad export cookie 607918531168267806 [ 5694.618390] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5694.626983] LustreError: 109598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5695.042653] Lustre: server umount lustre-MDT0001 complete [ 5709.894784] Lustre: server umount lustre-OST0000 complete [ 5713.696625] Lustre: 105833:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773612663/real 1773612663] req@ffffa0c105707100 x1859767302843776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773612679 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5713.728077] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5714.677550] Lustre: server umount lustre-OST0001 complete [ 5736.320679] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5750.535191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5751.076474] LustreError: 117257:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5751.098194] LustreError: 117257:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 5751.181168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5756.387319] LustreError: 117258:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5757.168912] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5761.514832] LustreError: 117257:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5766.629086] LustreError: 117258:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5769.879865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5770.273869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5775.870109] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5779.904862] Lustre: 118365:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5788.148601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5788.453705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5795.620545] LustreError: 118719:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5795.639255] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:33) [ 5796.139716] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5796.683220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:116 to 0x280000401:161) [ 5806.258662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5807.703565] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:33) [ 5807.736881] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 5813.938869] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5822.362866] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5826.125508] Lustre: 120208:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5845.660544] Lustre: DEBUG MARKER: == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 18:13:30 (1773612810) [ 5846.025476] Lustre: 117252:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 5846.037113] Lustre: 117252:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 46 previous similar messages [ 5846.044807] Lustre: 117252:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 5846.052309] Lustre: 117252:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 5846.059556] Lustre: 117252:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5846.069236] Lustre: 117252:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 5846.076457] Lustre: 117252:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 5846.083681] Lustre: 117252:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 5846.093903] Lustre: 117252:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 5846.098861] Lustre: 117252:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 5846.105769] Lustre: 117252:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5846.112450] Lustre: 117252:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 5848.105598] Lustre: *** cfs_fail_loc=1618, val=0*** [ 5848.119763] Lustre: Skipped 3 previous similar messages [ 5882.849604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5882.871441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5882.900343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5888.377108] Lustre: server umount lustre-MDT0000 complete [ 5888.481049] LustreError: 117254:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5888.497391] LustreError: 117254:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 5892.587073] LustreError: 117238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612858 with bad export cookie 607918531168283052 [ 5892.587761] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5892.598674] LustreError: 117238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5892.968602] Lustre: server umount lustre-MDT0001 complete [ 5907.643404] Lustre: server umount lustre-OST0000 complete [ 5909.471676] Lustre: 105833:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773612859/real 1773612859] req@ffffa0c005381180 x1859767302924544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773612875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5909.504866] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5909.527244] Lustre: Skipped 3 previous similar messages [ 5912.644565] Lustre: server umount lustre-OST0001 complete [ 5929.234655] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5939.856859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.430318] LustreError: 122761:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5940.504911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5940.510833] Lustre: Skipped 1 previous similar message [ 5945.937193] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5955.258042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.681180] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5962.905666] Lustre: 123870:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5969.931368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5976.163886] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5980.468464] LustreError: 124222:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5980.482094] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:65) [ 5980.486333] LustreError: 124222:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5983.578850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5984.812763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 5984.823090] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:65) [ 5984.920536] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 5989.839304] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5998.208824] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6002.364335] Lustre: 125714:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6013.085733] Lustre: 123791:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 252 < left 260, rollback = 2 [ 6013.104840] Lustre: 123791:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 21 previous similar messages [ 6013.115532] Lustre: 123791:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 6013.127698] Lustre: 123791:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 6013.133741] Lustre: 123791:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/260/0 [ 6013.141883] Lustre: 123791:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 6013.151863] Lustre: 123791:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/12/0, punch: 0/0/0, quota 4/150/2 [ 6013.159860] Lustre: 123791:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 6013.166320] Lustre: 123791:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 1/17/2, delete: 0/0/0 [ 6013.170031] Lustre: 123791:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 6013.174656] Lustre: 123791:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 6013.179455] Lustre: 123791:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 6013.215197] Lustre: *** cfs_fail_loc=1602, val=10*** [ 6041.504596] Lustre: DEBUG MARKER: == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 18:16:45 (1773613005) [ 6045.650593] Lustre: *** cfs_fail_loc=1616, val=0*** [ 6045.657563] Lustre: Skipped 3 previous similar messages [ 6052.172596] Lustre: *** cfs_fail_loc=161c, val=0*** [ 6071.167838] Lustre: DEBUG MARKER: == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) ========================================================== 18:17:15 (1773613035) [ 6073.286574] Lustre: *** cfs_fail_loc=162e, val=0*** [ 6087.984250] Lustre: DEBUG MARKER: == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 18:17:32 (1773613052) [ 6093.622774] Lustre: *** cfs_fail_loc=162f, val=0*** [ 6093.630076] Lustre: Skipped 9 previous similar messages [ 6096.147593] LustreError: 128758:0:(lfsck_layout.c:2103:lfsck_layout_add_comp()) five: Five five five five five five file Hello [0x280000401:0xc6:0x0] and [0x280000401:0xc6:0x0]d: rc = 0 [ 6110.784877] Lustre: DEBUG MARKER: == sanity-lfsck test 19a: OST-object inconsistency self detect ========================================================== 18:17:55 (1773613075) [ 6123.590263] Lustre: DEBUG MARKER: == sanity-lfsck test 19b: OST-object inconsistency self repair ========================================================== 18:18:08 (1773613088) [ 6126.636682] Lustre: *** cfs_fail_loc=1611, val=0*** [ 6126.673886] Lustre: *** cfs_fail_loc=1611, val=0*** [ 6126.679151] Lustre: Skipped 3 previous similar messages [ 6132.126307] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.14@tcp inode [0x2000013a1:0x1f:0x0] object 0x280000401:203 extent [0-4095], client returned csum 0 (type 4), server csum 703fb3ad (type 4) [ 6133.214440] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.14@tcp inode [0x2000013a1:0x20:0x0] object 0x280000401:204 extent [0-4095], client returned csum 0 (type 4), server csum 7cbb935c (type 4) [ 6141.051181] Lustre: DEBUG MARKER: == sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================================================== 18:18:25 (1773613105) [ 6141.414646] Lustre: 124518:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 6141.434633] Lustre: 124518:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 104 previous similar messages [ 6141.445583] Lustre: 124518:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 6141.454862] Lustre: 124518:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 6141.462407] Lustre: 124518:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 6141.469847] Lustre: 124518:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 6141.479971] Lustre: 124518:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 6141.487429] Lustre: 124518:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 6141.494384] Lustre: 124518:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 6141.502491] Lustre: 124518:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 6141.509430] Lustre: 124518:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 6141.518947] Lustre: 124518:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 6168.842652] Lustre: DEBUG MARKER: == sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case ========================================================== 18:18:53 (1773613133) [ 6176.453844] Lustre: DEBUG MARKER: == sanity-lfsck test 21: run all LFSCK components by default ========================================================== 18:19:00 (1773613140) [ 6192.016732] Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) ========================================================== 18:19:16 (1773613156) [ 6195.433983] Lustre: *** cfs_fail_loc=161e, val=0*** [ 6195.446728] Lustre: *** cfs_fail_loc=161e, val=0*** [ 6195.450472] Lustre: Skipped 1 previous similar message [ 6207.875657] Lustre: DEBUG MARKER: == sanity-lfsck test 22b: LFSCK can repair unmatched pairs (2) ========================================================== 18:19:32 (1773613172) [ 6209.902869] Lustre: *** cfs_fail_loc=161e, val=0*** [ 6209.911991] Lustre: *** cfs_fail_loc=161e, val=0*** [ 6221.835553] Lustre: DEBUG MARKER: == sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) ========================================================== 18:19:46 (1773613186) [ 6223.696450] Lustre: *** cfs_fail_loc=1620, val=0*** [ 6237.844487] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_23b skipping excluded test 23b [ 6239.539225] Lustre: DEBUG MARKER: == sanity-lfsck test 23c: LFSCK can repair dangling name entry (3) ========================================================== 18:20:04 (1773613204) [ 6247.968136] Lustre: *** cfs_fail_loc=1621, val=127*** [ 6247.974801] Lustre: Skipped 1 previous similar message [ 6251.924676] Lustre: *** cfs_fail_loc=1602, val=10*** [ 6272.800447] Lustre: DEBUG MARKER: == sanity-lfsck test 23d: LFSCK can repair a dangling name entry to a remote object ========================================================== 18:20:37 (1773613237) [ 6275.533487] Lustre: Failing over lustre-MDT0000 [ 6275.950940] Lustre: server umount lustre-MDT0000 complete [ 6276.577175] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6276.577474] LustreError: 130853:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6276.605031] Lustre: Skipped 2 previous similar messages [ 6276.624928] LustreError: 130853:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 6286.784061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6287.023236] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6287.376725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6287.389244] Lustre: Skipped 3 previous similar messages [ 6287.461069] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6291.930267] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6292.311829] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6292.467722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6292.511491] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6292.580416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 6292.587610] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:130 to 0x2c0000401:161) [ 6294.324709] LustreError: 122758:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0000: [0x2000013a3:0x82:0x0] doesn't exist!: rc = -14 [ 6305.031946] Lustre: DEBUG MARKER: == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ========================================================== 18:21:09 (1773613269) [ 6306.968551] Lustre: *** cfs_fail_loc=1622, val=0*** [ 6307.112433] Lustre: *** cfs_fail_loc=1622, val=0*** [ 6307.122021] Lustre: Skipped 1 previous similar message [ 6317.112076] Lustre: DEBUG MARKER: == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ========================================================== 18:21:21 (1773613281) [ 6318.775217] Lustre: *** cfs_fail_loc=1623, val=0*** [ 6329.771171] Lustre: DEBUG MARKER: == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace ========================================================== 18:21:34 (1773613294) [ 6331.348602] Lustre: *** cfs_fail_loc=1624, val=0*** [ 6342.209702] Lustre: DEBUG MARKER: == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========================================================== 18:21:46 (1773613306) [ 6355.938465] Lustre: DEBUG MARKER: == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan ========================================================== 18:22:00 (1773613320) [ 6357.641042] Lustre: *** cfs_fail_loc=1624, val=0*** [ 6357.643272] Lustre: Skipped 1 previous similar message [ 6368.433744] Lustre: DEBUG MARKER: == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ========================================================== 18:22:12 (1773613332) [ 6380.739983] Lustre: DEBUG MARKER: == sanity-lfsck test 28: Skip the failed MDT(s) when handle orphan MDT-objects ========================================================== 18:22:25 (1773613345) [ 6387.643594] Lustre: *** cfs_fail_loc=161c, val=0*** [ 6387.649705] Lustre: Skipped 2 previous similar messages [ 6414.560466] Lustre: DEBUG MARKER: == sanity-lfsck test 29b: LFSCK can repair bad nlink count (2) ========================================================== 18:22:58 (1773613378) [ 6416.957294] Lustre: 122756:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 6416.981458] Lustre: 122756:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 518 previous similar messages [ 6417.001837] Lustre: 122756:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 6417.025116] Lustre: 122756:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 518 previous similar messages [ 6417.045317] Lustre: 122756:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 6417.073567] Lustre: 122756:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 518 previous similar messages [ 6417.095797] Lustre: 122756:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 6417.133683] Lustre: 122756:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 518 previous similar messages [ 6417.172686] Lustre: 122756:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 6417.231163] Lustre: 122756:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 518 previous similar messages [ 6417.258891] Lustre: 122756:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 6417.284383] Lustre: 122756:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 518 previous similar messages [ 6421.815978] Lustre: *** cfs_fail_loc=1626, val=0*** [ 6421.821132] Lustre: Skipped 4 previous similar messages [ 6447.413490] Lustre: DEBUG MARKER: == sanity-lfsck test 29c: verify linkEA size limitation == 18:23:29 (1773613409) [ 6500.519993] Lustre: DEBUG MARKER: == sanity-lfsck test 29d: accessing non-existing inode shouldn't turn fs read-only (ldiskfs) ========================================================== 18:24:23 (1773613463) [ 6502.811804] Lustre: *** cfs_fail_loc=1626, val=0*** [ 6504.719329] LustreError: 122758:0:(osd_handler.c:284:osd_idc_find_or_init()) lustre-MDT0000: cannot lookup FID [0x2000013a3:0x9e:0x0]: rc = -2 [ 6516.776224] Lustre: DEBUG MARKER: == sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 18:24:40 (1773613480) [ 6530.271326] Lustre: Failing over lustre-MDT0000 [ 6531.067455] Lustre: server umount lustre-MDT0000 complete [ 6533.099824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6533.112228] LustreError: 140093:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6533.130287] Lustre: Skipped 2 previous similar messages [ 6533.158475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6533.160494] LustreError: 140093:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 12 previous similar messages [ 6549.463162] Lustre: 105833:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773613499/real 1773613499] req@ffffa0c004820000 x1859767303799680/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773613515 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6549.511875] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6550.510890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 6559.712035] LustreError: 105832:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0c003fb1c00 x1859767303808384/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6560.225213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.297670] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6565.367732] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 6565.398745] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6565.410127] Lustre: Skipped 3 previous similar messages [ 6565.423293] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 6565.462505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:295 to 0x280000401:321) [ 6565.465036] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:166 to 0x2c0000401:193) [ 6567.233375] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6585.942527] Lustre: DEBUG MARKER: == sanity-lfsck test 31a: The LFSCK can find/repair the name entry with bad name hash (1) ========================================================== 18:25:49 (1773613549) [ 6609.006405] Lustre: DEBUG MARKER: == sanity-lfsck test 31b: The LFSCK can find/repair the name entry with bad name hash (2) ========================================================== 18:26:12 (1773613572) [ 6632.740885] Lustre: DEBUG MARKER: == sanity-lfsck test 31c: Re-generate the lost master LMV EA for striped directory ========================================================== 18:26:36 (1773613596) [ 6635.037293] Lustre: *** cfs_fail_loc=1629, val=0*** [ 6635.039249] Lustre: Skipped 7 previous similar messages [ 6665.445979] Lustre: DEBUG MARKER: == sanity-lfsck test 31d: Set broken striped directory (modified after broken) as read-only ========================================================== 18:27:09 (1773613629) [ 6676.749871] Lustre: Failing over lustre-MDT0000 [ 6677.296653] Lustre: server umount lustre-MDT0000 complete [ 6677.986469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6677.990034] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6678.040623] Lustre: Skipped 5 previous similar messages [ 6689.660117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6689.829803] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6690.125601] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6695.400378] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 6695.424251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6695.447697] Lustre: Skipped 3 previous similar messages [ 6695.526494] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 6695.670876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:295 to 0x280000401:353) [ 6695.672908] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:166 to 0x2c0000401:225) [ 6695.683050] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6712.072828] Lustre: Failing over lustre-MDT0000 [ 6712.980725] Lustre: server umount lustre-MDT0000 complete [ 6727.360824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.601843] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6728.087863] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6731.244560] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6733.314987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6733.325491] Lustre: Skipped 3 previous similar messages [ 6733.365166] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6733.505736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:295 to 0x280000401:385) [ 6733.505811] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:227 to 0x2c0000401:257) [ 6735.067659] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6750.342279] Lustre: DEBUG MARKER: == sanity-lfsck test 31e: Re-generate the lost slave LMV EA for striped directory (1) ========================================================== 18:28:33 (1773613713) [ 6771.292818] Lustre: DEBUG MARKER: == sanity-lfsck test 31f: Re-generate the lost slave LMV EA for striped directory (2) ========================================================== 18:28:54 (1773613734) [ 6773.397850] Lustre: *** cfs_fail_loc=162a, val=1*** [ 6773.404186] Lustre: Skipped 5 previous similar messages [ 6792.781718] Lustre: DEBUG MARKER: == sanity-lfsck test 31g: Repair the corrupted slave LMV EA ========================================================== 18:29:16 (1773613756) [ 6838.132791] Lustre: DEBUG MARKER: == sanity-lfsck test 31h: Repair the corrupted shard's name entry ========================================================== 18:30:01 (1773613801) [ 6866.236831] Lustre: DEBUG MARKER: == sanity-lfsck test 32a: stop LFSCK when some OST failed ========================================================== 18:30:29 (1773613829) [ 6882.809028] LustreError: 147165:0:(lfsck_layout.c:4532:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 6885.813066] LustreError: 147165:0:(lfsck_layout.c:4532:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 6885.839661] LustreError: 147165:0:(lfsck_layout.c:4532:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 6887.342725] Lustre: Failing over lustre-OST0000 [ 6887.393773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6887.399754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6887.421946] Lustre: Skipped 4 previous similar messages [ 6887.427720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6887.442376] Lustre: Skipped 4 previous similar messages [ 6887.532435] Lustre: server umount lustre-OST0000 complete [ 6888.868685] LustreError: 147165:0:(lfsck_layout.c:4532:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 6888.878839] LustreError: 147165:0:(lfsck_layout.c:4532:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 6891.968112] LustreError: 147165:0:(lfsck_layout.c:4532:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 6891.992282] LustreError: 147165:0:(lfsck_layout.c:4532:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 6892.013051] LustreError: 125100:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6892.088709] LustreError: 125100:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 53 previous similar messages [ 6892.139529] LustreError: 147165:0:(lfsck_layout.c:4532:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout interrupted [ 6907.772209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 6908.232180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6908.246750] Lustre: Skipped 2 previous similar messages [ 6908.263864] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6909.955494] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6910.004858] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6910.011865] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6910.038923] Lustre: Skipped 3 previous similar messages [ 6917.844929] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6931.555332] Lustre: DEBUG MARKER: == sanity-lfsck test 32b: stop LFSCK when some MDT failed ========================================================== 18:31:35 (1773613895) [ 6956.252514] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6991.756847] Lustre: 149945:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7031.362964] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7038.175471] Lustre: 151082:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7047.427860] Lustre: 122758:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 7047.437268] Lustre: 122758:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 824 previous similar messages [ 7047.444473] Lustre: 122758:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 7047.451588] Lustre: 122758:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 824 previous similar messages [ 7047.459967] Lustre: 122758:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 7047.466226] Lustre: 122758:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 824 previous similar messages [ 7047.474056] Lustre: 122758:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 7047.480888] Lustre: 122758:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 824 previous similar messages [ 7047.486310] Lustre: 122758:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 7047.491458] Lustre: 122758:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 824 previous similar messages [ 7047.495943] Lustre: 122758:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 7047.503626] Lustre: 122758:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 824 previous similar messages [ 7067.801928] LustreError: 151216:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d sleeping for 3000ms [ 7070.888248] LustreError: 151216:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 7072.772273] Lustre: Failing over lustre-MDT0001 [ 7073.990429] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 7073.999991] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.022655] Lustre: Skipped 1 previous similar message [ 7074.029682] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7074.514675] Lustre: server umount lustre-MDT0001 complete [ 7097.594767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 7098.536312] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7104.007891] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7104.013271] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 7104.020941] Lustre: Skipped 1 previous similar message [ 7104.086183] Lustre: lustre-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 7104.207824] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:97) [ 7104.209923] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:97) [ 7106.869786] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7122.951069] Lustre: DEBUG MARKER: == sanity-lfsck test 33: check LFSCK paramters =========== 18:34:46 (1773614086) [ 7149.385958] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7183.983516] Lustre: 153911:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7224.284491] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7230.388863] Lustre: 155050:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7273.430185] Lustre: DEBUG MARKER: == sanity-lfsck test 34: LFSCK can rebuild the lost agent object ========================================================== 18:37:16 (1773614236) [ 7275.808404] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_34 Only valid for ZFS backend [ 7280.113971] Lustre: DEBUG MARKER: == sanity-lfsck test 35: LFSCK can rebuild the lost agent entry ========================================================== 18:37:23 (1773614243) [ 7294.877341] Lustre: *** cfs_fail_loc=1631, val=0*** [ 7313.383554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7313.414849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7318.610426] Lustre: server umount lustre-MDT0000 complete [ 7324.572376] LustreError: 122741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614290 with bad export cookie 607918531168355957 [ 7324.584768] LustreError: 122741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7324.590053] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7325.084611] Lustre: server umount lustre-MDT0001 complete [ 7342.694854] Lustre: server umount lustre-OST0000 complete [ 7345.645324] Lustre: 105834:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773614295/real 1773614295] req@ffffa0c1047bad80 x1859767304472704/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773614311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7345.693406] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7345.719370] Lustre: Skipped 6 previous similar messages [ 7348.996766] Lustre: server umount lustre-OST0001 complete [ 7380.226927] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7400.570352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7407.587642] LustreError: 157859:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7407.633814] LustreError: 157859:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 31 previous similar messages [ 7410.426381] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7426.151623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7426.597263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7426.610960] Lustre: Skipped 2 previous similar messages [ 7435.169782] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7440.525910] Lustre: 158967:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7451.844745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7462.800868] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7470.844375] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:129) [ 7471.886550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:542 to 0x280000401:577) [ 7477.744975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7479.662807] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:129) [ 7479.686481] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:414 to 0x2c0000401:449) [ 7489.805586] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7502.451806] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7508.215175] Lustre: 160814:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7530.390814] Lustre: DEBUG MARKER: == sanity-lfsck test 36a: rebuild LOV EA for mirrored file (1) ========================================================== 18:41:33 (1773614493) [ 7533.411522] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36a needs >= 3 OSTs [ 7536.236535] Lustre: DEBUG MARKER: == sanity-lfsck test 36b: rebuild LOV EA for mirrored file (2) ========================================================== 18:41:40 (1773614500) [ 7539.375619] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36b needs >= 3 OSTs [ 7542.754603] Lustre: DEBUG MARKER: == sanity-lfsck test 36c: rebuild LOV EA for mirrored file (3) ========================================================== 18:41:46 (1773614506) [ 7546.088350] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36c needs >= 3 OSTs [ 7550.025429] Lustre: DEBUG MARKER: == sanity-lfsck test 37: LFSCK must skip a ORPHAN ======== 18:41:53 (1773614513) [ 7572.343619] Lustre: DEBUG MARKER: == sanity-lfsck test 38: LFSCK does not break foreign file and reverse is also true ========================================================== 18:42:14 (1773614534) [ 7598.181523] Lustre: DEBUG MARKER: == sanity-lfsck test 39: LFSCK does not break foreign dir and reverse is also true ========================================================== 18:42:42 (1773614562) [ 7622.170214] Lustre: DEBUG MARKER: == sanity-lfsck test 40a: LFSCK correctly fixes lmm_oi in composite layout ========================================================== 18:43:05 (1773614585) [ 7650.692106] Lustre: DEBUG MARKER: == sanity-lfsck test 41: SEL support in LFSCK ============ 18:43:33 (1773614613) [ 7653.816772] Lustre: 157853:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 264, rollback = 2 [ 7653.842683] Lustre: 157853:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1196 previous similar messages [ 7653.870573] Lustre: 157853:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 7653.889868] Lustre: 157853:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1196 previous similar messages [ 7653.909920] Lustre: 157853:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 7653.932850] Lustre: 157853:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1196 previous similar messages [ 7653.948586] Lustre: 157853:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 7653.964081] Lustre: 157853:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1196 previous similar messages [ 7653.985774] Lustre: 157853:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 7653.996816] Lustre: 157853:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1196 previous similar messages [ 7654.007085] Lustre: 157853:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 7654.021885] Lustre: 157853:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1196 previous similar messages [ 7687.425729] Lustre: DEBUG MARKER: == sanity-lfsck test 42: LFSCK repairs inconsistent MDT-object/OST-object encryption flags ========================================================== 18:44:11 (1773614651) [ 7723.792792] Lustre: *** cfs_fail_loc=1632, val=0*** [ 7753.801049] Lustre: DEBUG MARKER: == sanity-lfsck test 43: LFSCK does not loop endlessly on iget failure in scanning-phase1 ========================================================== 18:45:15 (1773614715) [ 7758.329922] Lustre: Failing over lustre-MDT0001 [ 7758.411652] LustreError: 105836:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa0c1043b8000 x1859767304721280/t0(0) o103->lustre-MDT0000-osp-MDT0001@0@lo:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'jbd2/dm-1-8.0' uid:0 gid:0 projid:4294967295 [ 7759.004848] Lustre: server umount lustre-MDT0001 complete [ 7771.528797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7772.225331] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7772.226591] Lustre: lustre-MDT0001: Aborting client recovery [ 7772.259913] LustreError: 164634:0:(ldlm_lib.c:2983:target_stop_recovery_thread()) lustre-MDT0001: Aborting recovery [ 7772.284180] Lustre: 164662:0:(ldlm_lib.c:2386:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 7772.305624] Lustre: 164662:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 20b9e317-6c67-43cf-af3a-1b5f33b7e290@ [ 7772.328392] Lustre: lustre-MDT0001: disconnecting 2 stale clients [ 7772.371665] Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] [ 7772.413566] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] [ 7772.561265] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:161) [ 7772.567585] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7777.262764] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7777.277164] Lustre: Skipped 2 previous similar messages [ 7777.282967] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7779.376082] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7792.141747] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid [ 7792.769843] Lustre: DEBUG MARKER: osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid in FULL state after 0 sec [ 7806.203758] Lustre: DEBUG MARKER: == sanity-lfsck test 44: umount while lfsck is stopping == 18:46:09 (1773614769) [ 7822.129160] Lustre: *** cfs_fail_loc=1600, val=3*** [ 7827.574350] Lustre: Failing over lustre-MDT0000 [ 7828.450693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7828.925506] Lustre: server umount lustre-MDT0000 complete [ 7844.831175] Lustre: 105833:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773614794/real 1773614794] req@ffffa0c00eb11c00 x1859767304813824/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773614810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7844.886893] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7845.088064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7855.082372] LustreError: 105832:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0c00eb11880 x1859767304821760/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7855.764336] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7855.779948] Lustre: Skipped 2 previous similar messages [ 7856.825567] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7860.723986] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7860.735632] Lustre: Skipped 2 previous similar messages [ 7860.804904] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7860.884211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:621 to 0x280000401:641) [ 7860.901603] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:490 to 0x2c0000401:513) [ 7863.386479] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7881.568907] Lustre: DEBUG MARKER: == sanity-lfsck test 45: LFSCK should fix UID/GID/PROJID of OST object ========================================================== 18:47:25 (1773614845) [ 7923.762476] Lustre: Failing over lustre-OST0000 [ 7924.148027] Lustre: server umount lustre-OST0000 complete [ 7925.729111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7925.756594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7925.788545] Lustre: Skipped 7 previous similar messages [ 7934.869532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7954.198434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7954.652841] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 7955.870397] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7956.093862] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7956.097094] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7956.114564] Lustre: Skipped 3 previous similar messages [ 7964.028267] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7976.325863] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7976.642949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7984.172901] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7984.568216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7993.017339] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff9d1f4320a000.ost_server_uuid 50 [ 8054.574731] Lustre: DEBUG MARKER: rpc test_45: @@@@@@ FAIL: can't put import for osc.lustre-OST0000-osc-ffff9d1f4320a000.ost_server_uuid into FULL state after 50 sec, have IDLE