[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468065439 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] BRK [0x1f401000, 0x1f401fff] PGTABLE [ 0.000000] BRK [0x1f402000, 0x1f402fff] PGTABLE [ 0.000000] BRK [0x1f403000, 0x1f403fff] PGTABLE [ 0.000000] BRK [0x1f404000, 0x1f404fff] PGTABLE [ 0.000000] BRK [0x1f405000, 0x1f405fff] PGTABLE [ 0.000000] BRK [0x1f406000, 0x1f406fff] PGTABLE [ 0.000000] BRK [0x1f407000, 0x1f407fff] PGTABLE [ 0.000000] BRK [0x1f408000, 0x1f408fff] PGTABLE [ 0.000000] BRK [0x1f409000, 0x1f409fff] PGTABLE [ 0.000000] BRK [0x1f40a000, 0x1f40afff] PGTABLE [ 0.000000] BRK [0x1f40b000, 0x1f40bfff] PGTABLE [ 0.000000] BRK [0x1f40c000, 0x1f40cfff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] On node 0 totalpages: 1076588 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 158 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:63 [ 0.000000] Normal zone: 4536 pages used for memmap [ 0.000000] Normal zone: 290304 pages, LIFO batch:63 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] pcpu-alloc: s221184 r8192 d28672 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001000] hpet clockevent registered [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003389] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008549] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.012041] LSM: Security Framework initializing [ 0.013060] Yama: becoming mindful. [ 0.014055] SELinux: Initializing. [ 0.015105] *** VALIDATE selinux *** [ 0.023481] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027810] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] *** VALIDATE tmpfs *** [ 0.032321] *** VALIDATE proc *** [ 0.033242] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037047] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040013] Speculative Store Bypass: Vulnerable [ 0.042722] debug: unmapping init [mem 0xffffffffa4659000-0xffffffffa4660fff] [ 0.044148] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045648] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046028] ... version: 2 [ 0.047017] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049017] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051020] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.053520] rcu: Hierarchical SRCU implementation. [ 0.055493] smp: Bringing up secondary CPUs ... [ 0.056600] x86: Booting SMP configuration: [ 0.057033] .... node #0, CPUs: #1 #2 #3 [ 0.060636] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063030] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134684] node 0 deferred pages initialised in 69ms [ 0.137364] devtmpfs: initialized [ 0.139152] x86/mm: Memory block size: 128MB [ 0.141841] gcov: version magic: 0x41383552 [ 0.145274] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149342] pinctrl core: initialized pinctrl subsystem [ 0.151487] ************************************************************* [ 0.153125] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156020] ** ** [ 0.160023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162018] ** ** [ 0.165023] ** This means that this kernel is built to expose internal ** [ 0.168021] ** IOMMU data structures, which may compromise security on ** [ 0.171022] ** your system. ** [ 0.173020] ** ** [ 0.176020] ** If you see this message and you are not debugging the ** [ 0.179021] ** kernel, report this immediately to your vendor! ** [ 0.182020] ** ** [ 0.184019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187023] ************************************************************* [ 0.189774] NET: Registered protocol family 16 [ 0.192515] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195095] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.202091] cpuidle: using governor menu [ 0.205174] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206388] PCI: Using configuration type 1 for base access [ 0.209125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219099] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222054] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225146] cryptd: max_cpu_qlen set to 1000 [ 0.227107] ACPI: Added _OSI(Module Device) [ 0.228016] ACPI: Added _OSI(Processor Device) [ 0.229015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230031] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235343] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243035] ACPI: Interpreter enabled [ 0.245115] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247014] ACPI: Using IOAPIC for interrupt routing [ 0.250142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255381] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266786] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270069] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.283000] acpiphp: Slot [2] registered [ 0.284090] acpiphp: Slot [5] registered [ 0.286154] acpiphp: Slot [6] registered [ 0.288137] acpiphp: Slot [7] registered [ 0.289109] acpiphp: Slot [8] registered [ 0.290110] acpiphp: Slot [9] registered [ 0.292115] acpiphp: Slot [10] registered [ 0.293105] acpiphp: Slot [3] registered [ 0.295095] acpiphp: Slot [4] registered [ 0.296112] acpiphp: Slot [11] registered [ 0.297077] acpiphp: Slot [12] registered [ 0.299089] acpiphp: Slot [13] registered [ 0.300094] acpiphp: Slot [14] registered [ 0.301092] acpiphp: Slot [15] registered [ 0.303094] acpiphp: Slot [16] registered [ 0.304100] acpiphp: Slot [17] registered [ 0.305122] acpiphp: Slot [18] registered [ 0.307102] acpiphp: Slot [19] registered [ 0.308133] acpiphp: Slot [20] registered [ 0.310174] acpiphp: Slot [21] registered [ 0.312256] acpiphp: Slot [22] registered [ 0.313141] acpiphp: Slot [23] registered [ 0.314108] acpiphp: Slot [24] registered [ 0.316179] acpiphp: Slot [25] registered [ 0.318141] acpiphp: Slot [26] registered [ 0.320168] acpiphp: Slot [27] registered [ 0.321143] acpiphp: Slot [28] registered [ 0.323179] acpiphp: Slot [29] registered [ 0.325204] acpiphp: Slot [30] registered [ 0.327141] acpiphp: Slot [31] registered [ 0.328066] PCI host bridge to bus 0000:00 [ 0.329026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340052] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343047] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.347922] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351577] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.361816] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.367097] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.370022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.372015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.374015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.376585] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.379831] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.381037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384820] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.391014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.402016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.409015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.418230] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.427022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.436023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.453016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.465042] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.472013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.478014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.497019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.507196] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.516023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.527033] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.541028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.553696] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.561017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.567015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.586019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.596996] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.605018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.612021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.634022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.645588] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.651017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.658014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.671018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.683657] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.686414] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.689413] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.693492] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.696279] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.702151] iommu: Default domain type: Passthrough [ 0.705570] SCSI subsystem initialized [ 0.707195] ACPI: bus type USB registered [ 0.709133] usbcore: registered new interface driver usbfs [ 0.711133] usbcore: registered new interface driver hub [ 0.714150] usbcore: registered new device driver usb [ 0.716238] pps_core: LinuxPPS API ver. 1 registered [ 0.718019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.722091] PTP clock support registered [ 0.724135] EDAC MC: Ver: 3.0.0 [ 0.725581] PCI: Using ACPI for IRQ routing [ 0.726014] PCI: pci_cache_line_size set to 64 bytes [ 0.726406] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.726420] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.726423] e820: reserve RAM buffer [mem 0x146e00000-0x147ffffff] [ 0.726974] NetLabel: Initializing [ 0.727014] NetLabel: domain hash size = 128 [ 0.728011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.729091] NetLabel: unlabeled traffic allowed by default [ 0.731116] vgaarb: loaded [ 0.734192] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.736063] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.745237] clocksource: Switched to clocksource kvm-clock [ 0.854248] VFS: Disk quotas dquot_6.6.0 [ 0.856044] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.858795] *** VALIDATE ramfs *** [ 0.860226] *** VALIDATE hugetlbfs *** [ 0.862034] pnp: PnP ACPI init [ 0.863395] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.863498] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.863526] pnp 00:02: [dma 2] [ 0.863570] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.863625] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.863700] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.863776] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.864377] pnp: PnP ACPI: found 6 devices [ 0.881306] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.886155] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.888949] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.891657] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.893866] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.896568] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.899838] NET: Registered protocol family 2 [ 0.902461] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.907307] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.911156] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.917268] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.921702] TCP: Hash tables configured (established 65536 bind 65536) [ 0.924453] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.927900] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.931183] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.934630] NET: Registered protocol family 1 [ 0.937733] RPC: Registered named UNIX socket transport module. [ 0.940147] RPC: Registered udp transport module. [ 0.942046] RPC: Registered tcp transport module. [ 0.943741] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.946071] NET: Registered protocol family 44 [ 0.947415] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.949498] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.951353] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.953964] PCI: CLS 0 bytes, default 64 [ 0.956602] Unpacking initramfs... [ 2.355481] debug: unmapping init [mem 0xffff8c54bcc54000-0xffff8c54bffbffff] [ 2.359854] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.364274] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.367988] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.889329] Initialise system trusted keyrings [ 2.891426] Key type blacklist registered [ 2.893440] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.903759] zbud: loaded [ 2.907297] *** VALIDATE nfs *** [ 2.908775] *** VALIDATE nfs4 *** [ 2.911176] pstore: using deflate compression [ 2.916594] Platform Keyring initialized [ 3.027347] NET: Registered protocol family 38 [ 3.029373] Key type asymmetric registered [ 3.031183] Asymmetric key parser 'x509' registered [ 3.033263] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.036323] io scheduler mq-deadline registered [ 3.038279] io scheduler kyber registered [ 3.040225] io scheduler bfq registered [ 3.042270] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.045554] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.048784] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.051749] ACPI: Power Button [PWRF] [ 3.057548] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.064097] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.077642] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.084965] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.121258] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.150704] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.180897] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.187262] Non-volatile memory driver v1.3 [ 3.188762] Linux agpgart interface v0.103 [ 3.222949] virtio_blk virtio1: [vda] 145736 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.226477] vda: detected capacity change from 0 to 74616832 [ 3.242300] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.245823] vdb: detected capacity change from 0 to 1073741824 [ 3.272824] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.275803] vdc: detected capacity change from 0 to 2621440000 [ 3.289939] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.293246] vdd: detected capacity change from 0 to 2621440000 [ 3.307727] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.311236] vde: detected capacity change from 0 to 4294967296 [ 3.330268] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.333234] vdf: detected capacity change from 0 to 4294967296 [ 3.342716] libphy: Fixed MDIO Bus: probed [ 3.350251] usbcore: registered new interface driver usbserial_generic [ 3.352941] usbserial: USB Serial support registered for generic [ 3.355251] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.359514] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.361260] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.363476] mousedev: PS/2 mouse device common for all mice [ 3.365752] rtc_cmos 00:05: RTC can wake from S4 [ 3.369029] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.369505] rtc_cmos 00:05: registered as rtc0 [ 3.376055] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.376132] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.379053] intel_pstate: CPU model not supported [ 3.383980] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.389429] hid: raw HID events driver (C) Jiri Kosina [ 3.391763] usbcore: registered new interface driver usbhid [ 3.394378] usbhid: USB HID core driver [ 3.396068] drop_monitor: Initializing network drop monitor service [ 3.398757] Initializing XFRM netlink socket [ 3.400968] NET: Registered protocol family 10 [ 3.405125] Segment Routing with IPv6 [ 3.406754] NET: Registered protocol family 17 [ 3.408818] mpls_gso: MPLS GSO support [ 3.410579] start plist test [ 3.412424] end plist test [ 3.414026] RAS: Correctable Errors collector initialized. [ 3.416042] AVX version of gcm_enc/dec engaged. [ 3.417564] AES CTR mode by8 optimization enabled [ 3.491512] sched_clock: Marking stable (3491480053, 0)->(4433308837, -941828784) [ 3.495768] registered taskstats version 1 [ 3.497802] Loading compiled-in X.509 certificates [ 3.499942] zswap: loaded using pool lzo/zbud [ 3.523513] Key type big_key registered [ 3.536221] Key type encrypted registered [ 3.538218] ima: No TPM chip found, activating TPM-bypass! [ 3.540326] ima: Allocated hash algorithm: sha1 [ 3.542197] ima: No architecture policies found [ 3.543870] evm: Initialising EVM extended attributes: [ 3.545831] evm: security.selinux [ 3.546928] evm: security.ima [ 3.548166] evm: security.capability [ 3.549478] evm: HMAC attrs: 0x1 [ 3.551864] rtc_cmos 00:05: setting system clock to 2026-07-13 13:56:17 UTC (1783950977) [ 3.558153] debug: unmapping init [mem 0xffffffffa5603000-0xffffffffa57fffff] [ 3.561533] debug: unmapping init [mem 0xffffffffa4382000-0xffffffffa4658fff] [ 3.567087] Write protecting the kernel read-only data: 28672k [ 3.570652] debug: unmapping init [mem 0xffffffffa2a03000-0xffffffffa2bfffff] [ 3.573578] debug: unmapping init [mem 0xffffffffa3314000-0xffffffffa33fffff] [ 3.606786] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.616199] systemd[1]: Detected virtualization kvm. [ 3.618938] systemd[1]: Detected architecture x86-64. [ 3.620946] systemd[1]: Running in initial RAM disk. [ 3.652572] systemd[1]: No hostname configured. [ 3.654611] systemd[1]: Set hostname to . [ 3.656890] random: systemd: uninitialized urandom read (16 bytes read) [ 3.659350] systemd[1]: Initializing machine ID from random generator. [ 3.708212] random: ln: uninitialized urandom read (6 bytes read) [ 3.801643] random: systemd: uninitialized urandom read (16 bytes read) [ 3.804888] systemd[1]: Listening on Journal Socket (/dev/log). [ 3.810091] systemd[1]: Reached target Local File Systems. [ 3.814576] systemd[1]: Reached target Swap. [ 4.452928] device-mapper: uevent: version 1.0.3 [ 4.455977] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 5.222951] virtio_net virtio0 ens2: renamed from eth0 [ 5.226469] libata version 3.00 loaded. [ 5.230298] ata_piix 0000:00:01.1: version 2.13 [ 5.248250] random: fast init done [ 5.262262] scsi host0: ata_piix [ 5.283368] scsi host1: ata_piix [ 5.286559] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.290313] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.054241] random: crng init done [ 10.056448] random: 7 urandom warning(s) missed due to ratelimiting [ 10.527550] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 11.692995] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.940253] SELinux: Disabled at runtime. [ 12.002663] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.012072] systemd[1]: Detected virtualization kvm. [ 12.014248] systemd[1]: Detected architecture x86-64. [ 12.480795] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.484671] systemd[1]: Stopped Switch Root. [ 12.490495] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.504930] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.509039] systemd[1]: Stopped Journal Service. [ 12.516383] systemd[1]: Starting Journal Service... [ 12.524743] systemd[1]: Mounting Kernel Debug File System... [ 12.705831] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 12.933158] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 13.265632] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.331877] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.463241] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.474208] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 13.474233] EDAC sbridge: Ver: 1.1.2 [ 14.705613] Key type dns_resolver registered [ 15.015125] NFS: Registering the id_resolver key type [ 15.017224] Key type id_resolver registered [ 15.018946] Key type id_legacy registered [ 49.972840] libcfs: loading out-of-tree module taints kernel. [ 50.010489] Key type ._llcrypt registered [ 50.012277] Key type .llcrypt registered [ 50.128157] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 66.529322] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 67.694448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 67.711968] alg: No test for adler32 (adler32-zlib) [ 69.260430] Lustre: Lustre: Build Version: 2.17.54_162_gb5ded84 [ 70.240332] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 72.120411] Key type lgssc registered [ 74.489227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 92.071873] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 109.856077] hrtimer: interrupt took 4020504 ns [ 136.263086] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 148.370214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 148.399774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.680197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 149.718690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 149.783943] Lustre: lustre-MDT0000: new disk, initializing [ 149.879178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 149.901611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.799378] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.967737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.044272] Lustre: 6538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 165.070827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 165.074739] Lustre: Skipped 1 previous similar message [ 165.153557] Lustre: lustre-MDT0001: new disk, initializing [ 165.223548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 165.247127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 165.254384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.267226] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.791923] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 181.189904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 181.450646] Lustre: lustre-OST0000: new disk, initializing [ 181.454304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 181.459406] Lustre: 8473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 181.511269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.365519] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.979695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 188.995558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 189.037894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.903672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.107418] Lustre: lustre-OST0001: new disk, initializing [ 200.112883] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 200.137281] Lustre: 9545:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 200.237744] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 207.245650] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.385295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 207.390246] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 207.483361] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.672308] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.117304] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 232.674666] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.048789] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing yml_node [ 241.858794] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 244.435359] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 246.667297] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 248.152678] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Mon Jul 13 10:00:21 EDT 2026 [ 268.949664] Lustre: DEBUG MARKER: excepting tests: 18b 23b [ 280.817672] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 288.226764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 288.239386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.259421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.256408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.260167] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.280368] Lustre: Skipped 2 previous similar messages [ 294.026339] Lustre: server umount lustre-MDT0000 complete [ 299.490597] LustreError: 7855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.517632] LustreError: 7855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 301.954648] LustreError: 6530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783951275 with bad export cookie 14923208635077507719 [ 301.963285] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 301.973953] LustreError: 6530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 302.225102] Lustre: server umount lustre-MDT0001 complete [ 320.992168] Lustre: 3671:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783951278/real 1783951278] req@ffff8c5536a5ea00 x1870608250016256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783951294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 321.020563] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.204677] Lustre: server umount lustre-OST0000 complete [ 323.041111] Lustre: 3673:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783951280/real 1783951280] req@ffff8c5536a5f480 x1870608250016512/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783951296 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 325.153230] Lustre: 3671:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783951283/real 1783951283] req@ffff8c54035c9880 x1870608250016768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783951299 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 327.201414] Lustre: 3673:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783951285/real 1783951285] req@ffff8c5530ca1f80 x1870608250017280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783951301 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 327.231185] Lustre: 3673:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 330.605385] Lustre: server umount lustre-OST0001 complete [ 347.037696] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 350.021980] Key type lgssc unregistered [ 350.471385] LNet: 14830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.485446] LNetError: 14830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 350.513639] LNet: Removed LNI 192.168.203.117@tcp [ 351.491345] Key type .llcrypt unregistered [ 351.497092] Key type ._llcrypt unregistered [ 376.199404] Key type ._llcrypt registered [ 376.202729] Key type .llcrypt registered [ 376.316148] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 391.396692] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 392.163719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 392.281905] alg: No test for adler32 (adler32-zlib) [ 393.356693] Lustre: Lustre: Build Version: 2.17.54_162_gb5ded84 [ 393.721657] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 395.438707] Key type lgssc registered [ 397.106093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 450.721996] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 464.071302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 464.083037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.314072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 465.351374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 465.455913] Lustre: lustre-MDT0000: new disk, initializing [ 465.554814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.613840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 470.394868] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 484.332533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.408884] Lustre: 19288:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 484.439950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 484.448153] Lustre: Skipped 1 previous similar message [ 484.534952] Lustre: lustre-MDT0001: new disk, initializing [ 484.622148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 484.655333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 484.667477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 489.217325] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 493.842650] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 503.947568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 504.213295] Lustre: lustre-OST0000: new disk, initializing [ 504.216722] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 504.220533] Lustre: 21226:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 504.282887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 509.967072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 509.978607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 510.089551] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 510.493515] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 524.398089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 524.542432] Lustre: lustre-OST0001: new disk, initializing [ 524.546157] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 524.551380] Lustre: 22250:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 524.638116] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 530.145426] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 530.999414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 531.018123] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 536.051496] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 541.909954] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 550.400858] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 558.515792] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 10:05:31 (1783951531) === [ 561.159416] Lustre: DEBUG MARKER: == sanity-lfsck test 0: Control LFSCK manually =========== 10:05:34 (1783951534) [ 561.382837] Lustre: 22018:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 561.391501] Lustre: 22018:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 561.396626] Lustre: 22018:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 561.400773] Lustre: 22018:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 561.406691] Lustre: 22018:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 561.414392] Lustre: 22018:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 561.908841] Lustre: 22018:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 561.921398] Lustre: 22018:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 11 previous similar messages [ 561.931423] Lustre: 22018:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 561.936173] Lustre: 22018:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 561.941866] Lustre: 22018:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 561.948903] Lustre: 22018:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 561.960029] Lustre: 22018:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 561.971379] Lustre: 22018:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 561.979476] Lustre: 22018:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 561.986789] Lustre: 22018:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 561.994129] Lustre: 22018:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 562.000997] Lustre: 22018:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 562.982819] Lustre: 19294:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 562.991039] Lustre: 19294:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 53 previous similar messages [ 563.002075] Lustre: 19294:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 563.009821] Lustre: 19294:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 53 previous similar messages [ 563.021427] Lustre: 19294:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 563.030588] Lustre: 19294:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 53 previous similar messages [ 563.039189] Lustre: 19294:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 563.049600] Lustre: 19294:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 53 previous similar messages [ 563.057539] Lustre: 19294:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 563.066446] Lustre: 19294:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 53 previous similar messages [ 563.073116] Lustre: 19294:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 563.080816] Lustre: 19294:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 53 previous similar messages [ 565.036388] Lustre: 22018:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 565.052306] Lustre: 22018:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 107 previous similar messages [ 565.061077] Lustre: 22018:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 565.070324] Lustre: 22018:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 107 previous similar messages [ 565.079269] Lustre: 22018:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 565.094393] Lustre: 22018:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 107 previous similar messages [ 565.106214] Lustre: 22018:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 565.119472] Lustre: 22018:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 107 previous similar messages [ 565.129434] Lustre: 22018:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 565.136929] Lustre: 22018:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 107 previous similar messages [ 565.142375] Lustre: 22018:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 565.151802] Lustre: 22018:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 107 previous similar messages [ 569.071211] Lustre: *** cfs_fail_loc=1600, val=3*** [ 571.974048] Lustre: 21217:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 260 < left 274, rollback = 2 [ 571.989449] Lustre: 23650:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 571.992985] Lustre: 21217:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 104 previous similar messages [ 571.993040] Lustre: 21217:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/274/0 [ 571.993045] Lustre: 21217:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 103 previous similar messages [ 571.993051] Lustre: 21217:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 571.993055] Lustre: 21217:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 103 previous similar messages [ 571.993060] Lustre: 21217:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 571.993063] Lustre: 21217:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 103 previous similar messages [ 571.993067] Lustre: 21217:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 571.993070] Lustre: 21217:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 103 previous similar messages [ 572.126368] Lustre: 23650:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 112 previous similar messages [ 572.147787] Lustre: *** cfs_fail_loc=1600, val=3*** [ 574.253242] Lustre: *** cfs_fail_loc=1600, val=3*** [ 592.372133] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.375573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 592.400700] Lustre: Skipped 2 previous similar messages [ 592.409524] Lustre: Skipped 1 previous similar message [ 592.421811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 593.867846] Lustre: server umount lustre-MDT0000 complete [ 597.229613] LustreError: 19280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783951571 with bad export cookie 2661158562987455316 [ 597.233291] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.248655] LustreError: 19280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 597.527504] Lustre: server umount lustre-MDT0001 complete [ 611.785178] Lustre: server umount lustre-OST0000 complete [ 613.860996] Lustre: 16449:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783951571/real 1783951571] req@ffff8c540a194380 x1870608588713728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783951587 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 613.908559] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 613.938140] Lustre: Skipped 1 previous similar message [ 616.099307] Lustre: server umount lustre-OST0001 complete [ 626.035711] Lustre: DEBUG MARKER: == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 10:06:39 (1783951599) [ 641.057930] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 651.450725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.978559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 656.509291] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 657.376925] LustreError: 26307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.418265] LustreError: 26307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 661.474302] LustreError: 26306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.524042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 665.963905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 670.488808] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 673.278496] Lustre: 27447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 680.007554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 686.401777] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 690.533340] LustreError: 27799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 695.170169] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 695.513474] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 695.524682] Lustre: Skipped 1 previous similar message [ 696.565042] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:40 to 0x2c0000401:65) [ 696.565863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 702.348675] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 711.174815] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 715.453633] Lustre: 29318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 717.098486] Lustre: 26301:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 717.107991] Lustre: 26301:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 67 previous similar messages [ 717.114037] Lustre: 26301:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 717.124191] Lustre: 26301:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 59 previous similar messages [ 717.131518] Lustre: 26301:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 717.138753] Lustre: 26301:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 68 previous similar messages [ 717.150389] Lustre: 26301:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 717.159078] Lustre: 26301:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 68 previous similar messages [ 717.166345] Lustre: 26301:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 717.170924] Lustre: 26301:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 68 previous similar messages [ 717.176204] Lustre: 26301:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 717.182040] Lustre: 26301:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 68 previous similar messages [ 723.258961] Lustre: *** cfs_fail_loc=1501, val=0*** [ 732.015843] Lustre: Failing over lustre-MDT0000 [ 732.564149] Lustre: server umount lustre-MDT0000 complete [ 732.646441] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 732.650100] LustreError: 29085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 732.666372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 732.667092] Lustre: Skipped 2 previous similar messages [ 732.720329] LustreError: 29085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 737.765109] LustreError: 26302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 737.782166] LustreError: 26302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 742.900593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 742.976955] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 743.141579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 743.187190] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 747.478651] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 748.516545] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 748.527590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 748.553903] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 748.583762] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 748.589573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 750.903429] Lustre: *** cfs_fail_loc=1505, val=0*** [ 758.281276] Lustre: DEBUG MARKER: == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 10:08:51 (1783951731) [ 759.692498] Lustre: 26302:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 759.702471] Lustre: 26302:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 321 previous similar messages [ 759.710594] Lustre: 26302:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 759.719820] Lustre: 26302:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 759.726547] Lustre: 26302:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 759.735893] Lustre: 26302:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 759.745834] Lustre: 26302:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 759.760130] Lustre: 26302:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 759.771850] Lustre: 26302:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 759.778669] Lustre: 26302:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 759.786666] Lustre: 26302:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 759.794784] Lustre: 26302:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 765.262682] Lustre: *** cfs_fail_loc=1502, val=0*** [ 774.104659] Lustre: Failing over lustre-MDT0000 [ 774.117228] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.121657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -19 [ 774.140193] Lustre: Skipped 4 previous similar messages [ 774.142866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 774.176735] Lustre: Skipped 2 previous similar messages [ 774.456459] Lustre: server umount lustre-MDT0000 complete [ 779.237081] LustreError: 26306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 779.259936] LustreError: 26306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 785.236184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 785.349170] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.558499] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 790.064447] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 791.014242] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 791.020870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 791.041715] Lustre: Skipped 3 previous similar messages [ 791.060264] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 791.119040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 791.122487] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:169 to 0x2c0000401:193) [ 793.619507] Lustre: *** cfs_fail_loc=1505, val=0*** [ 801.042255] Lustre: DEBUG MARKER: == sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 10:09:34 (1783951774) [ 802.478477] Lustre: 26303:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 802.485171] Lustre: 26303:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 322 previous similar messages [ 802.495063] Lustre: 26303:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 802.501860] Lustre: 26303:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 802.507309] Lustre: 26303:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 802.511972] Lustre: 26303:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 802.518055] Lustre: 26303:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 802.522596] Lustre: 26303:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 802.526326] Lustre: 26303:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 802.534625] Lustre: 26303:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 802.541540] Lustre: 26303:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 802.551934] Lustre: 26303:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 808.295863] Lustre: *** cfs_fail_loc=1504, val=0*** [ 808.298262] Lustre: *** cfs_fail_loc=1504, val=0*** [ 808.303135] Lustre: Skipped 1 previous similar message [ 816.349127] Lustre: Failing over lustre-MDT0000 [ 816.546767] Lustre: server umount lustre-MDT0000 complete [ 816.609541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 816.633631] Lustre: Skipped 2 previous similar messages [ 816.639745] LustreError: 26302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 816.668025] LustreError: 26302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 827.978550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 828.098348] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 828.326538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.331497] Lustre: Skipped 1 previous similar message [ 828.375140] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 832.695300] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 833.511986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 833.516209] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 833.523899] Lustre: Skipped 3 previous similar messages [ 833.572341] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 833.619602] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:257) [ 833.620407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:233 to 0x280000401:257) [ 837.264730] Lustre: *** cfs_fail_loc=1505, val=0*** [ 848.014422] Lustre: DEBUG MARKER: == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 10:10:20 (1783951820) [ 856.121404] Lustre: *** cfs_fail_loc=1603, val=0*** [ 864.670841] Lustre: Failing over lustre-MDT0000 [ 864.846068] Lustre: server umount lustre-MDT0000 complete [ 869.345807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 869.364195] LustreError: 30780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.381700] Lustre: Skipped 3 previous similar messages [ 869.427596] LustreError: 30780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 869.451271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 876.549394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 876.626925] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 876.858185] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 881.308169] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 882.159771] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 882.174404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 882.183922] Lustre: Skipped 3 previous similar messages [ 882.196671] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 882.235413] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:297 to 0x2c0000401:321) [ 882.239336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:297 to 0x280000401:321) [ 890.052297] Lustre: DEBUG MARKER: == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 10:11:03 (1783951863) [ 891.289542] Lustre: 26302:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 891.297725] Lustre: 26302:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 644 previous similar messages [ 891.304638] Lustre: 26302:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 891.310289] Lustre: 26302:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 891.315949] Lustre: 26302:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 891.324989] Lustre: 26302:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 891.330249] Lustre: 26302:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 891.336411] Lustre: 26302:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 891.341649] Lustre: 26302:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 891.346148] Lustre: 26302:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 891.350694] Lustre: 26302:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 891.355798] Lustre: 26302:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 896.998637] Lustre: *** cfs_fail_loc=1604, val=0*** [ 904.271250] Lustre: Failing over lustre-MDT0000 [ 904.449557] Lustre: server umount lustre-MDT0000 complete [ 907.747779] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.765545] Lustre: Skipped 1 previous similar message [ 907.770149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 914.523420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 914.614600] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 914.836603] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 919.026892] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 920.037835] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 920.041707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 920.059558] Lustre: Skipped 3 previous similar messages [ 920.081048] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 920.124487] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:361 to 0x2c0000401:385) [ 920.124928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:361 to 0x280000401:385) [ 928.731245] Lustre: DEBUG MARKER: == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 10:11:42 (1783951902) [ 936.170536] Lustre: *** cfs_fail_loc=1605, val=0*** [ 944.005980] Lustre: Failing over lustre-MDT0000 [ 944.417398] Lustre: server umount lustre-MDT0000 complete [ 945.649299] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 945.651869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 945.652380] LustreError: 26302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 945.652388] LustreError: 26302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 945.676981] Lustre: Skipped 2 previous similar messages [ 955.894789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 956.058498] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 956.308593] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 961.507071] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 961.516267] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 961.524941] Lustre: Skipped 3 previous similar messages [ 961.549435] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 961.611548] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:425 to 0x2c0000401:449) [ 961.612548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:425 to 0x280000401:449) [ 961.620354] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 972.179388] Lustre: DEBUG MARKER: == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 10:12:25 (1783951945) [ 979.705796] Lustre: *** cfs_fail_loc=161d, val=0*** [ 987.836544] Lustre: Failing over lustre-MDT0000 [ 988.300240] Lustre: server umount lustre-MDT0000 complete [ 992.226791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 998.359531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 998.473212] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 998.665692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.673722] Lustre: Skipped 3 previous similar messages [ 998.710150] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1003.175383] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1004.013195] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1004.018808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1004.047447] Lustre: Skipped 3 previous similar messages [ 1004.076950] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1004.130414] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 1004.137085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:489 to 0x280000401:513) [ 1013.564898] Lustre: DEBUG MARKER: == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 10:13:06 (1783951986) [ 1016.371164] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1026.726542] Lustre: DEBUG MARKER: == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 10:13:20 (1783952000) [ 1027.341116] Lustre: 26302:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 1027.348631] Lustre: 26302:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 976 previous similar messages [ 1027.354859] Lustre: 26302:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1027.363694] Lustre: 26302:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 976 previous similar messages [ 1027.380969] Lustre: 26302:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1027.387360] Lustre: 26302:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 976 previous similar messages [ 1027.402871] Lustre: 26302:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1027.419960] Lustre: 26302:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 976 previous similar messages [ 1027.433912] Lustre: 26302:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 1027.441835] Lustre: 26302:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 976 previous similar messages [ 1027.453226] Lustre: 26302:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1027.465599] Lustre: 26302:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 976 previous similar messages [ 1033.666849] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1034.551821] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1046.138937] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 10:13:39 (1783952019) [ 1082.096511] Lustre: Failing over lustre-MDT0000 [ 1082.490863] Lustre: server umount lustre-MDT0000 complete [ 1085.928646] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.931092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1085.936158] LustreError: 26301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.936170] LustreError: 26301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 1085.955270] Lustre: Skipped 5 previous similar messages [ 1088.255462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1098.292451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1101.216653] Lustre: 16447:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952059/real 1783952059] req@ffff8c553a8b6680 x1870608589358080/t0(0) o400->MGC192.168.203.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783952075 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1101.277744] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.901679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1108.920727] Lustre: lustre-MDT0000: reset Object Index mappings [ 1111.520659] LustreError: 16446:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8c55075f7800 x1870608589365376/t0(0) o250->MGC192.168.203.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1111.978573] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1115.856077] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1117.152800] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1117.156904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1117.171152] Lustre: Skipped 3 previous similar messages [ 1117.198503] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1117.275497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:609) [ 1117.276102] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:609) [ 1119.439402] LustreError: 42958:0:(lfsck_engine.c:1018:lfsck_master_engine()) lustre-MDT0000-osd: master engine fail to verify the .lustre/lost+found/, go ahead: rc = -115 [ 1119.463852] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1123.552674] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1123.560381] Lustre: Skipped 3 previous similar messages [ 1128.937736] Lustre: Failing over lustre-MDT0000 [ 1129.288400] Lustre: server umount lustre-MDT0000 complete [ 1140.335329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1145.223477] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1145.942071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:641) [ 1145.942202] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:641) [ 1148.982954] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1158.381696] Lustre: DEBUG MARKER: == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 10:15:31 (1783952131) [ 1161.384235] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1171.536684] Lustre: Failing over lustre-MDT0000 [ 1171.818737] Lustre: server umount lustre-MDT0000 complete [ 1177.711430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1189.784076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1192.934053] Lustre: 16447:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952150/real 1783952150] req@ffff8c553aa31880 x1870608589455744/t0(0) o400->MGC192.168.203.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783952166 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1204.717811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1204.744210] Lustre: lustre-MDT0000: reset Object Index mappings [ 1217.865664] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1217.884054] Lustre: Skipped 1 previous similar message [ 1222.103911] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1223.149130] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1223.156818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1223.165182] Lustre: Skipped 1 previous similar message [ 1223.186500] Lustre: Skipped 7 previous similar messages [ 1223.220228] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1223.230248] Lustre: Skipped 1 previous similar message [ 1223.291587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:705) [ 1223.294665] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:705) [ 1226.488729] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1239.272520] Lustre: Failing over lustre-MDT0000 [ 1239.657431] Lustre: server umount lustre-MDT0000 complete [ 1243.618607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1243.622112] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.629813] LustreError: Skipped 1 previous similar message [ 1243.646428] Lustre: Skipped 13 previous similar messages [ 1251.700379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1251.824238] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1251.835372] LustreError: Skipped 2 previous similar messages [ 1257.326361] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1257.494648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:737) [ 1257.519863] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:737) [ 1261.038523] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1261.041500] Lustre: Skipped 84 previous similar messages [ 1269.892540] Lustre: DEBUG MARKER: == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 10:17:23 (1783952243) [ 1280.532703] Lustre: *** cfs_fail_loc=1600, val=1*** [ 1280.541192] Lustre: Skipped 7 previous similar messages [ 1302.042882] Lustre: DEBUG MARKER: == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 10:17:54 (1783952274) [ 1303.665476] Lustre: 30780:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 1303.672965] Lustre: 30780:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1355 previous similar messages [ 1303.678040] Lustre: 30780:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1303.684325] Lustre: 30780:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1355 previous similar messages [ 1303.688887] Lustre: 30780:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1303.693618] Lustre: 30780:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1355 previous similar messages [ 1303.697642] Lustre: 30780:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1303.706243] Lustre: 30780:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1355 previous similar messages [ 1303.711915] Lustre: 30780:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 1303.715039] Lustre: 30780:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1355 previous similar messages [ 1303.722315] Lustre: 30780:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1303.726686] Lustre: 30780:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1355 previous similar messages [ 1313.056279] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1313.059052] Lustre: Skipped 11 previous similar messages [ 1338.645367] Lustre: DEBUG MARKER: == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 10:18:31 (1783952311) [ 1358.902835] Lustre: Failing over lustre-MDT0000 [ 1359.267269] Lustre: server umount lustre-MDT0000 complete [ 1359.841799] LustreError: 26307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.857394] LustreError: 26307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 81 previous similar messages [ 1369.292733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1369.787173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1369.792533] Lustre: Skipped 4 previous similar messages [ 1369.839828] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1369.852210] Lustre: Skipped 1 previous similar message [ 1375.209357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1375.213151] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1375.225026] Lustre: Skipped 7 previous similar messages [ 1375.259020] Lustre: Skipped 1 previous similar message [ 1375.274689] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1375.281523] Lustre: Skipped 1 previous similar message [ 1375.414083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:853 to 0x280000401:897) [ 1375.417073] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:854 to 0x2c0000401:897) [ 1375.448846] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1386.308109] Lustre: DEBUG MARKER: == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 10:19:19 (1783952359) [ 1402.595505] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1421.279367] Lustre: 53000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1443.871835] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1447.380888] Lustre: 54136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1455.436956] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1455.450138] Lustre: Skipped 81 previous similar messages [ 1458.165363] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1459.233456] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1460.256884] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1462.304876] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1462.309394] Lustre: Skipped 1 previous similar message [ 1463.234906] Lustre: Failing over lustre-MDT0000 [ 1463.663197] Lustre: server umount lustre-MDT0000 complete [ 1475.218610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1481.276795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:947 to 0x2c0000401:993) [ 1481.278806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:946 to 0x280000401:961) [ 1481.722626] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1492.731730] Lustre: DEBUG MARKER: == sanity-lfsck test 8: LFSCK state machine ============== 10:21:05 (1783952465) [ 1496.546020] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1496.550569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.562217] LustreError: Skipped 1 previous similar message [ 1496.593019] Lustre: Skipped 3 previous similar messages [ 1501.666624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.679565] Lustre: Skipped 3 previous similar messages [ 1506.786455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1506.791343] Lustre: Skipped 3 previous similar messages [ 1509.344277] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1509.520773] Lustre: server umount lustre-MDT0000 complete [ 1513.633657] LustreError: 27824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783952487 with bad export cookie 2661158562987670069 [ 1513.654691] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1513.668283] LustreError: 27824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1513.699425] LustreError: Skipped 2 previous similar messages [ 1514.184786] Lustre: server umount lustre-MDT0001 complete [ 1528.523611] Lustre: server umount lustre-OST0000 complete [ 1542.929692] Lustre: server umount lustre-OST0001 complete [ 1551.041670] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 1560.452626] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1604.653759] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1615.140167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.464685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1615.506211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1615.566270] Lustre: lustre-MDT0000: new disk, initializing [ 1615.653432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1621.000732] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1632.742357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.893755] Lustre: 59200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1632.938675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1632.951203] Lustre: Skipped 1 previous similar message [ 1633.076128] Lustre: lustre-MDT0001: new disk, initializing [ 1633.233053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1633.244115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1639.257393] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1644.088666] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1651.824864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1651.948404] Lustre: lustre-OST0000: new disk, initializing [ 1651.951972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1651.956786] Lustre: 60833:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1653.156923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1653.177224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1653.227973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1658.523354] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1670.185986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1670.333640] Lustre: lustre-OST0001: new disk, initializing [ 1670.338561] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1670.344751] Lustre: 61705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1672.016199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1672.030203] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1672.120432] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1678.219657] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1688.425610] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1692.948496] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1704.461244] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1705.645637] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1705.651672] Lustre: Skipped 19 previous similar messages [ 1710.661035] Lustre: *** cfs_fail_loc=1601, val=2*** [ 1710.664313] Lustre: Skipped 17 previous similar messages [ 1727.542213] Lustre: Failing over lustre-MDT0000 [ 1727.932982] Lustre: server umount lustre-MDT0000 complete [ 1728.498450] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1728.520877] Lustre: Skipped 15 previous similar messages [ 1738.480525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1738.839715] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1738.849857] Lustre: Skipped 1 previous similar message [ 1743.846928] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1743.850714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1743.876627] Lustre: Skipped 1 previous similar message [ 1743.900126] Lustre: Skipped 7 previous similar messages [ 1743.958213] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1743.969235] Lustre: Skipped 1 previous similar message [ 1744.007189] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1744.018649] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:65) [ 1744.020587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:65) [ 1744.885348] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1753.951211] Lustre: Failing over lustre-MDT0000 [ 1754.080458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1754.099896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1754.112893] Lustre: Skipped 1 previous similar message [ 1754.484755] Lustre: server umount lustre-MDT0000 complete [ 1765.178893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1769.436890] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1771.032154] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1771.038834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:97) [ 1771.047347] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:97) [ 1775.762781] Lustre: Failing over lustre-MDT0000 [ 1776.133497] Lustre: server umount lustre-MDT0000 complete [ 1786.234645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1791.871876] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1792.064362] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:129) [ 1792.072371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:129) [ 1798.576377] Lustre: *** cfs_fail_loc=1602, val=2*** [ 1814.450143] Lustre: DEBUG MARKER: == sanity-lfsck test 9a: LFSCK speed control (1) ========= 10:26:27 (1783952787) [ 1829.747712] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1847.526827] Lustre: 68659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1869.450196] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1874.075324] Lustre: 69796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1884.531476] Lustre: 59208:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1884.561015] Lustre: 59208:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 2445 previous similar messages [ 1884.583272] Lustre: 59208:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1884.610274] Lustre: 59208:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 2445 previous similar messages [ 1884.632933] Lustre: 59208:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1884.652861] Lustre: 59208:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 2445 previous similar messages [ 1884.668959] Lustre: 59208:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1884.682657] Lustre: 59208:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 2445 previous similar messages [ 1884.696225] Lustre: 59208:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1884.710867] Lustre: 59208:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 2445 previous similar messages [ 1884.718547] Lustre: 59208:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1884.726503] Lustre: 59208:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 2445 previous similar messages [ 1993.376987] Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) ========= 10:29:26 (1783952966) [ 2046.009348] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2046.016898] Lustre: Skipped 4 previous similar messages [ 2069.654043] Lustre: *** cfs_fail_loc=160c, val=0*** [ 2069.656658] Lustre: Skipped 7 previous similar messages [ 2108.593648] Lustre: DEBUG MARKER: == sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 10:31:21 (1783953081) [ 2164.351115] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2165.401852] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2165.404675] Lustre: Skipped 43 previous similar messages [ 2167.455977] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2167.457816] Lustre: Skipped 111 previous similar messages [ 2171.464710] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2171.469392] Lustre: Skipped 220 previous similar messages [ 2179.476898] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2179.478937] Lustre: Skipped 414 previous similar messages [ 2195.504131] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2195.509706] Lustre: Skipped 740 previous similar messages [ 2227.519651] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2227.525107] Lustre: Skipped 1569 previous similar messages [ 2236.037654] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2236.045411] Lustre: Skipped 2599 previous similar messages [ 2477.764775] Lustre: DEBUG MARKER: == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 10:37:31 (1783953451) [ 2583.412943] Lustre: 59208:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 320, rollback = 2 [ 2583.423122] Lustre: 59208:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 36831 previous similar messages [ 2583.429855] Lustre: 59208:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 1/4/0 [ 2583.436084] Lustre: 59208:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2583.446611] Lustre: 59208:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 5/5/0, xattr_set: 7/320/0 [ 2583.456779] Lustre: 59208:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2583.465384] Lustre: 59208:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 6/46/0, punch: 0/0/0, quota 1/3/0 [ 2583.472683] Lustre: 59208:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2583.482699] Lustre: 59208:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 2/5/0 [ 2583.488781] Lustre: 59208:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2583.496847] Lustre: 59208:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 2583.501643] Lustre: 59208:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2652.129223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2652.130779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2652.131487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.152673] Lustre: Skipped 10 previous similar messages [ 2652.184725] LustreError: Skipped 1 previous similar message [ 2657.249429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.257414] Lustre: Skipped 5 previous similar messages [ 2658.086248] Lustre: server umount lustre-MDT0000 complete [ 2661.760969] LustreError: 61715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783953635 with bad export cookie 2661158562987689172 [ 2661.764946] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2661.771988] LustreError: 61715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2661.781752] LustreError: Skipped 3 previous similar messages [ 2662.081325] Lustre: server umount lustre-MDT0001 complete [ 2675.382264] Lustre: server umount lustre-OST0000 complete [ 2678.758933] Lustre: 16450:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783953636/real 1783953636] req@ffff8c553f1c1880 x1870608593605632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783953652 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2679.100276] Lustre: server umount lustre-OST0001 complete [ 2685.700213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 2694.609959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2710.240924] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2715.361439] LustreError: 75074:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.117@tcp: failed processing log, type 4: rc = -110 [ 2741.088205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2741.094544] Lustre: Skipped 8 previous similar messages [ 2748.170377] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2752.788668] Lustre: 75657:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2752.807744] Lustre: *** cfs_fail_loc=160e, val=3*** [ 2755.908235] Lustre: 75657:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2765.383143] Lustre: DEBUG MARKER: == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 10:42:18 (1783953738) [ 2780.249480] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2791.165398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.639100] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.663545] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 2791.788230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3271 to 0x280000401:3297) [ 2797.204532] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2806.060818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.344704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2806.353312] Lustre: Skipped 1 previous similar message [ 2811.200202] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2814.404280] Lustre: 78275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2832.404182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2838.012968] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3208 to 0x2c0000401:3233) [ 2840.656459] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2848.472179] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2852.296201] Lustre: 79774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2857.174403] Lustre: *** cfs_fail_loc=160d, val=0*** [ 2862.999640] Lustre: Failing over lustre-OST0000 [ 2863.081381] LustreError: 79287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.094749] LustreError: 79287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [ 2863.109391] Lustre: server umount lustre-OST0000 complete [ 2874.840096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.033319] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2875.046785] Lustre: Skipped 2 previous similar messages [ 2876.521426] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2876.533489] Lustre: Skipped 2 previous similar messages [ 2876.554247] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2876.558804] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2876.560858] Lustre: *** cfs_fail_loc=215, val=0*** [ 2876.567866] Lustre: Skipped 2 previous similar messages [ 2876.584439] Lustre: Skipped 11 previous similar messages [ 2881.819741] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2882.030722] Lustre: *** cfs_fail_loc=215, val=0*** [ 2882.033777] Lustre: Skipped 2 previous similar messages [ 2886.323062] Lustre: 81173:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2886.338606] Lustre: 81173:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2887.141533] Lustre: *** cfs_fail_loc=215, val=0*** [ 2887.145056] Lustre: Skipped 3 previous similar messages [ 2889.287983] Lustre: Failing over lustre-OST0000 [ 2889.398966] Lustre: server umount lustre-OST0000 complete [ 2898.126254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2899.572306] Lustre: *** cfs_fail_loc=215, val=0*** [ 2904.687965] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2905.057302] Lustre: *** cfs_fail_loc=215, val=0*** [ 2905.066894] Lustre: Skipped 2 previous similar messages [ 2913.769162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2913.775136] Lustre: Skipped 3 previous similar messages [ 2916.724980] Lustre: server umount lustre-MDT0000 complete [ 2920.280179] LustreError: 75081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783953894 with bad export cookie 2661158562989257578 [ 2920.282175] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2920.291702] LustreError: 75081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2920.625328] Lustre: server umount lustre-MDT0001 complete [ 2935.027226] Lustre: server umount lustre-OST0000 complete [ 2948.997590] Lustre: server umount lustre-OST0001 complete [ 2958.690421] Lustre: DEBUG MARKER: == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 10:45:31 (1783953931) [ 2975.243416] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2985.047819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.473127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2985.478775] Lustre: Skipped 3 previous similar messages [ 2990.218920] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2995.681863] LustreError: 84418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.707194] LustreError: 84418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 2998.958295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.528875] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3007.524690] Lustre: 85558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3014.629127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3022.908393] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3030.508624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3362 to 0x280000401:3393) [ 3033.111967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3038.701346] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3208 to 0x2c0000401:3265) [ 3039.877451] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3046.867878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3051.009586] Lustre: 87426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3087.655969] Lustre: DEBUG MARKER: == sanity-lfsck test 12b: auto detect Lustre device ====== 10:47:40 (1783954060) [ 3103.551481] Lustre: DEBUG MARKER: == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 10:47:57 (1783954077) [ 3104.944637] Lustre: *** cfs_fail_loc=160f, val=0*** [ 3116.931935] Lustre: DEBUG MARKER: == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 10:48:10 (1783954090) [ 3120.918102] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3120.919710] Lustre: Skipped 7 previous similar messages [ 3171.809980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3176.860977] Lustre: server umount lustre-MDT0000 complete [ 3180.663160] LustreError: 84398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954154 with bad export cookie 2661158562989266027 [ 3180.678285] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3180.683949] LustreError: 84398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3181.044262] Lustre: server umount lustre-MDT0001 complete [ 3195.629221] Lustre: server umount lustre-OST0000 complete [ 3197.602823] Lustre: 16447:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783954155/real 1783954155] req@ffff8c553fb01f80 x1870608593939840/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783954171 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3199.404563] Lustre: server umount lustre-OST0001 complete [ 3219.180612] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3231.605881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.574496] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3245.520311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3245.821963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3245.824943] Lustre: Skipped 4 previous similar messages [ 3249.989355] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3253.010651] Lustre: 93291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3259.898482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.708626] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3270.435400] LustreError: 93649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.456057] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:161) [ 3270.458902] LustreError: 93649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3274.791993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3277.997734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3554 to 0x280000401:3585) [ 3278.020233] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:161) [ 3278.076581] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3361) [ 3280.736387] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3288.646449] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3292.682435] Lustre: 95163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3299.473945] Lustre: DEBUG MARKER: == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 10:51:12 (1783954272) [ 3301.504669] Lustre: 92148:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 3301.522201] Lustre: 92148:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1371 previous similar messages [ 3301.533804] Lustre: 92148:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 3301.543472] Lustre: 92148:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1371 previous similar messages [ 3301.555908] Lustre: 92148:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3301.561879] Lustre: 92148:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1371 previous similar messages [ 3301.571461] Lustre: 92148:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3301.580775] Lustre: 92148:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1371 previous similar messages [ 3301.587410] Lustre: 92148:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 3301.595019] Lustre: 92148:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1371 previous similar messages [ 3301.605459] Lustre: 92148:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3301.613340] Lustre: 92148:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1371 previous similar messages [ 3304.732362] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3304.743346] Lustre: Skipped 63 previous similar messages [ 3327.968863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3327.976749] LustreError: Skipped 5 previous similar messages [ 3327.996351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.022642] Lustre: Skipped 15 previous similar messages [ 3328.030969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3328.033951] Lustre: Skipped 3 previous similar messages [ 3332.456782] Lustre: server umount lustre-MDT0000 complete [ 3336.529655] LustreError: 95165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954310 with bad export cookie 2661158562989294440 [ 3336.543023] LustreError: 95165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3336.836550] Lustre: server umount lustre-MDT0001 complete [ 3350.926118] Lustre: server umount lustre-OST0000 complete [ 3366.175086] Lustre: server umount lustre-OST0001 complete [ 3383.587541] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3394.313419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.324301] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3408.174400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.134693] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3417.527113] Lustre: 99202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3425.022243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3432.310519] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3433.662520] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:193) [ 3440.137706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3682 to 0x280000401:3713) [ 3441.757658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3447.274285] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:193) [ 3447.278267] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3393) [ 3448.228557] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3456.016100] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3466.969951] Lustre: DEBUG MARKER: == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 10:54:00 (1783954440) [ 3470.892387] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3470.898686] Lustre: Skipped 63 previous similar messages [ 3471.117408] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3482.626108] Lustre: DEBUG MARKER: == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 10:54:15 (1783954455) [ 3485.084857] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3485.092489] Lustre: Skipped 1 previous similar message [ 3485.162722] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3485.169108] Lustre: Skipped 2 previous similar messages [ 3496.589382] Lustre: DEBUG MARKER: == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 10:54:29 (1783954469) [ 3498.143837] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_15c MDS newer than 2.7.55, LU-6475 [ 3499.895973] Lustre: DEBUG MARKER: == sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 10:54:33 (1783954473) [ 3506.911230] Lustre: *** cfs_fail_loc=1709, val=0*** [ 3506.924280] LustreError: 98071:0:(mdt_reint.c:2644:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240002340:0x6c:0x0]/f2 failed: rc = -5 [ 3577.317692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3577.322895] Lustre: Skipped 4 previous similar messages [ 3583.434130] Lustre: server umount lustre-MDT0000 complete [ 3591.404939] LustreError: 98044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954565 with bad export cookie 2661158562989309175 [ 3591.413188] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3591.415121] LustreError: 98044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3591.428961] LustreError: Skipped 1 previous similar message [ 3591.739129] Lustre: server umount lustre-MDT0001 complete [ 3610.450887] Lustre: server umount lustre-OST0000 complete [ 3611.618599] Lustre: 16448:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783954569/real 1783954569] req@ffff8c553eb74000 x1870608594708608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783954585 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3616.931335] Lustre: 16450:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783954574/real 1783954574] req@ffff8c553e7a0a80 x1870608594709120/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783954590 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3616.963082] Lustre: 16450:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3618.968930] Lustre: server umount lustre-OST0001 complete [ 3637.508686] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 3640.435242] Key type lgssc unregistered [ 3640.788305] LNet: 104914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3640.799462] LNetError: 104914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3640.837248] LNet: Removed LNI 192.168.203.117@tcp [ 3641.697136] Key type .llcrypt unregistered [ 3641.701454] Key type ._llcrypt unregistered [ 3668.564635] Key type ._llcrypt registered [ 3668.576018] Key type .llcrypt registered [ 3668.708316] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 3681.709519] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3682.711894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3682.822517] alg: No test for adler32 (adler32-zlib) [ 3683.970530] Lustre: Lustre: Build Version: 2.17.54_162_gb5ded84 [ 3684.232434] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 3685.890928] Key type lgssc registered [ 3687.264711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3738.285118] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3751.545144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3751.590603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.848236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3752.895990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3752.976440] Lustre: lustre-MDT0000: new disk, initializing [ 3753.052075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3753.069575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3757.666578] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3771.420881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3771.533832] Lustre: 109350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3771.572457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3771.584100] Lustre: Skipped 1 previous similar message [ 3771.670450] Lustre: lustre-MDT0001: new disk, initializing [ 3771.753209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3771.782733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3771.794312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3776.664445] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3781.975916] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3791.893504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3792.231824] Lustre: lustre-OST0000: new disk, initializing [ 3792.235568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3792.243630] Lustre: 111289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3792.307906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3798.744785] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3802.188184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3802.197696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3802.317308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3812.797860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3812.941611] Lustre: lustre-OST0001: new disk, initializing [ 3812.944181] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3812.948394] Lustre: 112314:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3812.999237] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3819.678944] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3821.617639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3821.638076] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3821.672641] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3832.392210] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3839.185502] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3846.200734] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 11:00:18 (1783954818) === [ 3854.555826] Lustre: DEBUG MARKER: == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 11:00:27 (1783954827) [ 3854.806338] Lustre: 113305:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 3854.830060] Lustre: 113305:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3854.845092] Lustre: 113305:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3854.861813] Lustre: 113305:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 3854.876968] Lustre: 113305:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 3854.888570] Lustre: 113305:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3855.352106] Lustre: 109358:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 3855.370448] Lustre: 109358:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 6 previous similar messages [ 3855.378140] Lustre: 109358:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 3855.394377] Lustre: 109358:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 3855.407985] Lustre: 109358:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3855.418901] Lustre: 109358:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 3855.426273] Lustre: 109358:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3855.441773] Lustre: 109358:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 3855.453342] Lustre: 109358:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 3855.475500] Lustre: 109358:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 3855.483033] Lustre: 109358:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3855.489402] Lustre: 109358:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 3856.357670] Lustre: 109359:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 3856.369445] Lustre: 109359:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 113 previous similar messages [ 3856.381952] Lustre: 109359:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 3856.394912] Lustre: 109359:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 113 previous similar messages [ 3856.411978] Lustre: 109359:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3856.419686] Lustre: 109359:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 113 previous similar messages [ 3856.427430] Lustre: 109359:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 7/369/0 [ 3856.433515] Lustre: 109359:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 115 previous similar messages [ 3856.460908] Lustre: 109360:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 3856.466583] Lustre: 109360:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 119 previous similar messages [ 3856.492892] Lustre: 109358:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3856.497963] Lustre: 109358:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 3858.852943] Lustre: *** cfs_fail_loc=1613, val=0*** [ 3871.254818] Lustre: DEBUG MARKER: == sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 11:00:44 (1783954844) [ 3872.616670] Lustre: 113305:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 3872.628825] Lustre: 113305:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 188 previous similar messages [ 3872.639353] Lustre: 113305:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 3872.644405] Lustre: 113305:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 188 previous similar messages [ 3872.649986] Lustre: 113305:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3872.659869] Lustre: 113305:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 188 previous similar messages [ 3872.667251] Lustre: 113305:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3872.674962] Lustre: 113305:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 186 previous similar messages [ 3872.681506] Lustre: 113305:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 3872.688495] Lustre: 113305:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 182 previous similar messages [ 3872.694654] Lustre: 113305:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3872.713791] Lustre: 113305:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 176 previous similar messages [ 3873.824345] Lustre: *** cfs_fail_loc=1614, val=0*** [ 3874.331669] Lustre: *** cfs_fail_loc=1614, val=103*** [ 3874.336398] Lustre: Skipped 1 previous similar message [ 3879.119325] Lustre: 111280:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 275, rollback = 2 [ 3879.126370] Lustre: 111280:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 11 previous similar messages [ 3879.137508] Lustre: 111280:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3879.147583] Lustre: 111280:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3879.166025] Lustre: 111280:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 3/275/0 [ 3879.184372] Lustre: 111280:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3879.194793] Lustre: 111280:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 1/4/0, quota 7/225/0 [ 3879.201825] Lustre: 111280:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3879.219492] Lustre: 111280:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 3879.234417] Lustre: 111280:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3879.242819] Lustre: 111280:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3879.253865] Lustre: 111280:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3888.954657] Lustre: DEBUG MARKER: == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 11:01:02 (1783954862) [ 3889.261308] Lustre: 109358:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 3889.277967] Lustre: 109358:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 3889.287063] Lustre: 109358:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3889.293090] Lustre: 109358:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3889.305929] Lustre: 109358:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3889.321494] Lustre: 109358:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3889.330338] Lustre: 109358:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 3889.337172] Lustre: 109358:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3889.346491] Lustre: 109358:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 3889.353519] Lustre: 109358:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3889.359601] Lustre: 109358:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3889.364079] Lustre: 109358:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3892.204355] Lustre: *** cfs_fail_loc=1615, val=0*** [ 3892.213310] Lustre: Skipped 1 previous similar message [ 3893.396850] Lustre: *** cfs_fail_loc=1615, val=0*** [ 3893.398749] Lustre: Skipped 1 previous similar message [ 3911.713646] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_18b skipping excluded test 18b [ 3913.513836] Lustre: DEBUG MARKER: == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 11:01:26 (1783954886) [ 3913.917816] Lustre: 109358:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 3913.924799] Lustre: 109358:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 21 previous similar messages [ 3913.929624] Lustre: 109358:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 3913.934176] Lustre: 109358:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3913.943709] Lustre: 109358:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3913.952877] Lustre: 109358:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3913.962132] Lustre: 109358:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3913.972043] Lustre: 109358:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3913.980230] Lustre: 109358:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 3913.987348] Lustre: 109358:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3913.996167] Lustre: 109358:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3914.003739] Lustre: 109358:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3916.243820] Lustre: *** cfs_fail_loc=1617, val=0*** [ 3916.354164] Lustre: *** cfs_fail_loc=1617, val=0*** [ 3918.917282] Lustre: *** cfs_fail_loc=1616, val=0*** [ 3918.919396] Lustre: Skipped 3 previous similar messages [ 3937.242916] Lustre: DEBUG MARKER: == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 11:01:50 (1783954910) [ 3939.750252] Lustre: *** cfs_fail_loc=1618, val=0*** [ 3939.753519] Lustre: Skipped 5 previous similar messages [ 3975.144708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3975.155102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3975.165056] Lustre: Skipped 1 previous similar message [ 3979.592361] Lustre: server umount lustre-MDT0000 complete [ 3983.347989] LustreError: 110766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954957 with bad export cookie 3176701766802664599 [ 3983.348769] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3983.364048] LustreError: 110766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3983.599448] Lustre: server umount lustre-MDT0001 complete [ 3997.309846] Lustre: server umount lustre-OST0000 complete [ 4011.501877] Lustre: server umount lustre-OST0001 complete [ 4028.639503] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 4039.632847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4039.929713] LustreError: 118040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.943985] LustreError: 118040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4039.991385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4044.682610] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4045.285867] LustreError: 118041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4049.379612] LustreError: 118040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4053.378423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4053.576065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4058.319945] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4061.533803] Lustre: 119179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4069.841744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4070.289335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4076.743760] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4078.498222] LustreError: 119531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.539330] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:33) [ 4083.685601] LustreError: 119974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.718147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:116 to 0x280000401:161) [ 4086.029654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4091.382499] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 4091.398995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:33) [ 4092.996880] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4100.715611] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4104.196409] Lustre: 121051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4118.085229] Lustre: DEBUG MARKER: == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 11:04:51 (1783955091) [ 4118.567312] Lustre: 120818:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 4118.575509] Lustre: 120818:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 46 previous similar messages [ 4118.585502] Lustre: 120818:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 4118.591263] Lustre: 120818:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4118.598753] Lustre: 120818:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4118.608742] Lustre: 120818:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4118.618653] Lustre: 120818:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4118.627304] Lustre: 120818:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4118.635691] Lustre: 120818:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 4118.645097] Lustre: 120818:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4118.672641] Lustre: 120818:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4118.680895] Lustre: 120818:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4120.857299] Lustre: *** cfs_fail_loc=1618, val=0*** [ 4120.861028] Lustre: Skipped 3 previous similar messages [ 4155.873972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4155.883526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4155.897733] Lustre: Skipped 2 previous similar messages [ 4155.903407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4155.906262] Lustre: Skipped 3 previous similar messages [ 4157.922052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4157.930377] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.933866] Lustre: Skipped 2 previous similar messages [ 4157.971064] Lustre: Skipped 2 previous similar messages [ 4160.875947] Lustre: server umount lustre-MDT0000 complete [ 4163.043029] LustreError: 118040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4163.067802] LustreError: 118040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4165.450229] LustreError: 118021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955139 with bad export cookie 3176701766802679838 [ 4165.460282] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4165.478226] LustreError: 118021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4165.898574] Lustre: server umount lustre-MDT0001 complete [ 4180.629054] Lustre: server umount lustre-OST0000 complete [ 4193.621275] Lustre: server umount lustre-OST0001 complete [ 4215.022482] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 4226.581311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.139972] LustreError: 123624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.249059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4227.256543] Lustre: Skipped 1 previous similar message [ 4232.057757] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4241.608704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4248.845949] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4252.853449] Lustre: 124764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4260.319219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4267.395764] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4267.750970] LustreError: 125121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4267.784040] LustreError: 125121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4267.785496] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:65) [ 4272.878709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4276.827795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4282.350825] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:65) [ 4282.358183] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 4283.523242] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4291.413870] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4294.815571] Lustre: 126636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4300.425945] Lustre: 123625:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 252 < left 260, rollback = 2 [ 4300.431454] Lustre: 123625:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 21 previous similar messages [ 4300.437285] Lustre: 123625:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4300.440607] Lustre: 123625:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4300.444602] Lustre: 123625:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/260/0 [ 4300.448977] Lustre: 123625:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4300.453709] Lustre: 123625:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 0/0/0, quota 4/150/2 [ 4300.461756] Lustre: 123625:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4300.467572] Lustre: 123625:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 1/17/2, delete: 0/0/0 [ 4300.474476] Lustre: 123625:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4300.484251] Lustre: 123625:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4300.490541] Lustre: 123625:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4300.516289] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4324.867713] Lustre: DEBUG MARKER: == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 11:08:18 (1783955298) [ 4328.271468] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4328.276203] Lustre: Skipped 3 previous similar messages [ 4334.918601] Lustre: *** cfs_fail_loc=161c, val=0*** [ 4352.949604] Lustre: DEBUG MARKER: == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) ========================================================== 11:08:46 (1783955326) [ 4354.867157] Lustre: *** cfs_fail_loc=162e, val=0*** [ 4368.059804] Lustre: DEBUG MARKER: == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 11:09:01 (1783955341) [ 4373.342107] Lustre: *** cfs_fail_loc=162f, val=0*** [ 4373.343854] Lustre: Skipped 9 previous similar messages [ 4389.766256] Lustre: DEBUG MARKER: == sanity-lfsck test 19a: OST-object inconsistency self detect ========================================================== 11:09:22 (1783955362) [ 4403.480918] Lustre: DEBUG MARKER: == sanity-lfsck test 19b: OST-object inconsistency self repair ========================================================== 11:09:36 (1783955376) [ 4407.011395] Lustre: *** cfs_fail_loc=1611, val=0*** [ 4407.052289] Lustre: *** cfs_fail_loc=1611, val=0*** [ 4407.054596] Lustre: Skipped 3 previous similar messages [ 4412.488599] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.17@tcp inode [0x2000013a1:0x1f:0x0] object 0x280000401:203 extent [0-4095], client returned csum 0 (type 4), server csum 703fb3ad (type 4) [ 4413.630416] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.17@tcp inode [0x2000013a1:0x20:0x0] object 0x280000401:204 extent [0-4095], client returned csum 0 (type 4), server csum 7cbb935c (type 4) [ 4422.182536] Lustre: DEBUG MARKER: == sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================================================== 11:09:55 (1783955395) [ 4432.417128] Lustre: 131143:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 263, rollback = 2 [ 4432.446344] Lustre: 131143:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 125 previous similar messages [ 4432.470093] Lustre: 131143:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4432.481690] Lustre: 131143:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4432.515331] Lustre: 131143:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 2/263/0 [ 4432.534607] Lustre: 131143:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4432.543069] Lustre: 131143:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/2 [ 4432.557340] Lustre: 131143:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4432.574428] Lustre: 131143:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 4432.586655] Lustre: 131143:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4432.599742] Lustre: 131143:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4432.609077] Lustre: 131143:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4448.341950] Lustre: DEBUG MARKER: == sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case ========================================================== 11:10:21 (1783955421) [ 4456.206700] Lustre: DEBUG MARKER: == sanity-lfsck test 21: run all LFSCK components by default ========================================================== 11:10:29 (1783955429) [ 4470.123263] Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) ========================================================== 11:10:43 (1783955443) [ 4472.808832] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4472.820953] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4472.825838] Lustre: Skipped 1 previous similar message [ 4484.216415] Lustre: DEBUG MARKER: == sanity-lfsck test 22b: LFSCK can repair unmatched pairs (2) ========================================================== 11:10:57 (1783955457) [ 4485.829504] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4485.833511] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4497.087761] Lustre: DEBUG MARKER: == sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) ========================================================== 11:11:10 (1783955470) [ 4498.906995] Lustre: *** cfs_fail_loc=1620, val=0*** [ 4513.741574] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_23b skipping excluded test 23b [ 4515.498747] Lustre: DEBUG MARKER: == sanity-lfsck test 23c: LFSCK can repair dangling name entry (3) ========================================================== 11:11:28 (1783955488) [ 4521.859371] Lustre: *** cfs_fail_loc=1621, val=127*** [ 4521.864859] Lustre: Skipped 1 previous similar message [ 4524.807653] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4547.432358] Lustre: DEBUG MARKER: == sanity-lfsck test 23d: LFSCK can repair a dangling name entry to a remote object ========================================================== 11:12:00 (1783955520) [ 4550.202039] Lustre: Failing over lustre-MDT0000 [ 4550.647443] Lustre: server umount lustre-MDT0000 complete [ 4550.772889] LustreError: 123619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4550.796350] LustreError: 123619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4553.710787] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4553.750176] Lustre: Skipped 2 previous similar messages [ 4561.212516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.382925] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.555916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.564209] Lustre: Skipped 3 previous similar messages [ 4561.600427] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4565.378187] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4566.079127] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4567.023568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4567.057768] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4567.104978] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:130 to 0x2c0000401:161) [ 4567.109560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 4567.476431] LustreError: 126225:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0000: [0x2000013a3:0x82:0x0] doesn't exist!: rc = -14 [ 4578.382871] Lustre: DEBUG MARKER: == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ========================================================== 11:12:31 (1783955551) [ 4580.343794] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4580.536164] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4580.540443] Lustre: Skipped 1 previous similar message [ 4592.154560] Lustre: DEBUG MARKER: == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ========================================================== 11:12:45 (1783955565) [ 4593.789504] Lustre: *** cfs_fail_loc=1623, val=0*** [ 4605.374082] Lustre: DEBUG MARKER: == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace ========================================================== 11:12:58 (1783955578) [ 4607.334639] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4618.308687] Lustre: DEBUG MARKER: == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========================================================== 11:13:11 (1783955591) [ 4632.823721] Lustre: DEBUG MARKER: == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan ========================================================== 11:13:25 (1783955605) [ 4634.680618] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4634.682642] Lustre: Skipped 1 previous similar message [ 4646.881311] Lustre: DEBUG MARKER: == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ========================================================== 11:13:39 (1783955619) [ 4661.620351] Lustre: DEBUG MARKER: == sanity-lfsck test 28: Skip the failed MDT(s) when handle orphan MDT-objects ========================================================== 11:13:54 (1783955634) [ 4668.040088] Lustre: *** cfs_fail_loc=161c, val=0*** [ 4668.045073] Lustre: Skipped 1 previous similar message [ 4683.643707] Lustre: DEBUG MARKER: == sanity-lfsck test 29b: LFSCK can repair bad nlink count (2) ========================================================== 11:14:17 (1783955657) [ 4686.036943] Lustre: *** cfs_fail_loc=1626, val=0*** [ 4686.038838] Lustre: Skipped 4 previous similar messages [ 4696.818550] Lustre: DEBUG MARKER: == sanity-lfsck test 29c: verify linkEA size limitation == 11:14:30 (1783955670) [ 4697.259938] Lustre: 123621:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 320, rollback = 2 [ 4697.267513] Lustre: 123621:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 508 previous similar messages [ 4697.277028] Lustre: 123621:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 1/4/0 [ 4697.281383] Lustre: 123621:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 508 previous similar messages [ 4697.288751] Lustre: 123621:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 5/5/1, xattr_set: 7/320/0 [ 4697.297537] Lustre: 123621:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 508 previous similar messages [ 4697.302285] Lustre: 123621:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 6/46/0, punch: 0/0/0, quota 1/3/0 [ 4697.307307] Lustre: 123621:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 508 previous similar messages [ 4697.311770] Lustre: 123621:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 2/5/1 [ 4697.316198] Lustre: 123621:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 508 previous similar messages [ 4697.320714] Lustre: 123621:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1 [ 4697.324812] Lustre: 123621:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 508 previous similar messages [ 4726.493310] Lustre: DEBUG MARKER: == sanity-lfsck test 29d: accessing non-existing inode shouldn't turn fs read-only (ldiskfs) ========================================================== 11:14:59 (1783955699) [ 4729.364825] LustreError: 123621:0:(osd_handler.c:284:osd_idc_find_or_init()) lustre-MDT0000: cannot lookup FID [0x2000013a3:0x9e:0x0]: rc = -2 [ 4736.178843] Lustre: DEBUG MARKER: == sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 11:15:09 (1783955709) [ 4759.461061] Lustre: Failing over lustre-MDT0000 [ 4759.763491] Lustre: server umount lustre-MDT0000 complete [ 4761.568639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4761.572595] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4761.575780] LustreError: 123621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.590505] Lustre: Skipped 4 previous similar messages [ 4761.615769] LustreError: 123621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 4769.068943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4769.180157] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4769.369927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.421048] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4773.062768] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4774.372681] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4774.377618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4774.391119] Lustre: Skipped 3 previous similar messages [ 4774.418627] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4774.471630] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:166 to 0x2c0000401:193) [ 4774.473444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:295 to 0x280000401:321) [ 4783.890489] Lustre: DEBUG MARKER: == sanity-lfsck test 31a: The LFSCK can find/repair the name entry with bad name hash (1) ========================================================== 11:15:57 (1783955757) [ 4796.264366] Lustre: DEBUG MARKER: == sanity-lfsck test 31b: The LFSCK can find/repair the name entry with bad name hash (2) ========================================================== 11:16:09 (1783955769) [ 4810.226301] Lustre: DEBUG MARKER: == sanity-lfsck test 31c: Re-generate the lost master LMV EA for striped directory ========================================================== 11:16:23 (1783955783) [ 4811.557152] Lustre: *** cfs_fail_loc=1629, val=0*** [ 4811.560647] Lustre: Skipped 7 previous similar messages [ 4823.590707] Lustre: DEBUG MARKER: == sanity-lfsck test 31d: Set broken striped directory (modified after broken) as read-only ========================================================== 11:16:36 (1783955796) [ 4829.695985] Lustre: Failing over lustre-MDT0000 [ 4829.965632] Lustre: server umount lustre-MDT0000 complete [ 4830.695294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.713881] Lustre: Skipped 2 previous similar messages [ 4830.722182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4838.258898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4838.333847] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4838.639975] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4842.900575] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4844.013543] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4844.026988] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4844.062148] Lustre: Skipped 3 previous similar messages [ 4844.096738] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4844.153689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:295 to 0x280000401:353) [ 4844.153980] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:166 to 0x2c0000401:225) [ 4853.288735] Lustre: Failing over lustre-MDT0000 [ 4853.685514] Lustre: server umount lustre-MDT0000 complete [ 4854.240610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4854.247314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4854.256872] Lustre: Skipped 1 previous similar message [ 4861.964128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4862.108599] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4862.367979] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4866.108322] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4866.739957] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4867.569407] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4867.580056] Lustre: Skipped 3 previous similar messages [ 4867.647667] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4867.692349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:295 to 0x280000401:385) [ 4867.692454] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:227 to 0x2c0000401:257) [ 4875.270315] Lustre: DEBUG MARKER: == sanity-lfsck test 31e: Re-generate the lost slave LMV EA for striped directory (1) ========================================================== 11:17:28 (1783955848) [ 4886.273315] Lustre: DEBUG MARKER: == sanity-lfsck test 31f: Re-generate the lost slave LMV EA for striped directory (2) ========================================================== 11:17:39 (1783955859) [ 4898.525654] Lustre: DEBUG MARKER: == sanity-lfsck test 31g: Repair the corrupted slave LMV EA ========================================================== 11:17:51 (1783955871) [ 4938.103660] Lustre: DEBUG MARKER: == sanity-lfsck test 31h: Repair the corrupted shard's name entry ========================================================== 11:18:31 (1783955911) [ 4953.197119] Lustre: DEBUG MARKER: == sanity-lfsck test 32a: stop LFSCK when some OST failed ========================================================== 11:18:46 (1783955926) [ 4964.124299] LustreError: 148033:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 4966.987343] Lustre: Failing over lustre-OST0000 [ 4967.082933] Lustre: server umount lustre-OST0000 complete [ 4967.220549] LustreError: 148033:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 4967.231955] LustreError: 148033:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 4969.441159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4969.453769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.472366] Lustre: Skipped 3 previous similar messages [ 4970.096794] LustreError: 148033:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout interrupted [ 4980.877521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4981.335043] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4983.152260] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4983.185906] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4983.185952] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4983.211677] Lustre: Skipped 3 previous similar messages [ 4988.957501] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4997.797345] Lustre: DEBUG MARKER: == sanity-lfsck test 32b: stop LFSCK when some MDT failed ========================================================== 11:19:30 (1783955970) [ 5012.443982] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 5029.099136] Lustre: 150836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5050.200177] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5053.722316] Lustre: 151969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5066.709718] LustreError: 152109:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5069.680612] Lustre: Failing over lustre-MDT0001 [ 5069.800437] LustreError: 152109:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 5069.802817] LustreError: 152108:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5069.805550] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 5069.805572] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5069.805575] Lustre: Skipped 1 previous similar message [ 5069.805933] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5069.805937] Lustre: Skipped 1 previous similar message [ 5069.807710] LustreError: 152109:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) Skipped 1 previous similar message [ 5069.911695] LustreError: 152108:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) Skipped 1 previous similar message [ 5070.180105] Lustre: server umount lustre-MDT0001 complete [ 5072.904516] LustreError: 152108:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 5073.378821] LustreError: 126225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5073.387698] LustreError: 126225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 30 previous similar messages [ 5084.198518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5084.469586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5084.475900] Lustre: Skipped 3 previous similar messages [ 5084.502232] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5088.782451] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5089.762182] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5089.763997] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5089.774680] Lustre: Skipped 1 previous similar message [ 5089.792796] Lustre: lustre-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5089.884442] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:97) [ 5089.905445] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:97) [ 5097.711944] Lustre: DEBUG MARKER: == sanity-lfsck test 33: check LFSCK paramters =========== 11:21:11 (1783956071) [ 5111.082564] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 5128.026279] Lustre: 154828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5151.190471] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5155.238154] Lustre: 155962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5177.405101] Lustre: DEBUG MARKER: == sanity-lfsck test 34: LFSCK can rebuild the lost agent object ========================================================== 11:22:30 (1783956150) [ 5178.903494] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_34 Only valid for ZFS backend [ 5180.664620] Lustre: DEBUG MARKER: == sanity-lfsck test 35: LFSCK can rebuild the lost agent entry ========================================================== 11:22:33 (1783956153) [ 5188.208062] Lustre: *** cfs_fail_loc=1631, val=0*** [ 5202.402884] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5202.411864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5202.411991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5202.422100] Lustre: Skipped 5 previous similar messages [ 5202.435840] Lustre: Skipped 3 previous similar messages [ 5206.385728] Lustre: server umount lustre-MDT0000 complete [ 5210.717460] LustreError: 124766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783956184 with bad export cookie 3176701766802752666 [ 5210.726436] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5210.732623] LustreError: 124766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5210.984837] Lustre: server umount lustre-MDT0001 complete [ 5225.221207] Lustre: server umount lustre-OST0000 complete [ 5228.832488] Lustre: 106511:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783956186/real 1783956186] req@ffff8c540aa82680 x1870612041142656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783956202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5229.506642] Lustre: server umount lustre-OST0001 complete [ 5244.959611] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 5253.399653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5258.188455] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5265.890913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.133818] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5272.562795] Lustre: 159854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5277.789247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5279.147198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:540 to 0x280000401:577) [ 5283.562584] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5291.513393] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:129) [ 5293.286369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5298.685220] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:129) [ 5298.692631] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:412 to 0x2c0000401:449) [ 5299.613589] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5307.257605] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5310.663868] Lustre: 161724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5319.328978] Lustre: DEBUG MARKER: == sanity-lfsck test 36a: rebuild LOV EA for mirrored file (1) ========================================================== 11:24:52 (1783956292) [ 5320.959476] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36a needs >= 3 OSTs [ 5322.910978] Lustre: DEBUG MARKER: == sanity-lfsck test 36b: rebuild LOV EA for mirrored file (2) ========================================================== 11:24:56 (1783956296) [ 5324.783805] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36b needs >= 3 OSTs [ 5327.233467] Lustre: DEBUG MARKER: == sanity-lfsck test 36c: rebuild LOV EA for mirrored file (3) ========================================================== 11:25:00 (1783956300) [ 5328.883883] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36c needs >= 3 OSTs [ 5330.835836] Lustre: DEBUG MARKER: == sanity-lfsck test 37: LFSCK must skip a ORPHAN ======== 11:25:04 (1783956304) [ 5332.553112] Lustre: 158708:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 5332.559594] Lustre: 158708:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1919 previous similar messages [ 5332.565295] Lustre: 158708:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 5332.570202] Lustre: 158708:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1919 previous similar messages [ 5332.574771] Lustre: 158708:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5332.579319] Lustre: 158708:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1919 previous similar messages [ 5332.584459] Lustre: 158708:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 5332.589526] Lustre: 158708:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1919 previous similar messages [ 5332.595856] Lustre: 158708:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 5332.600343] Lustre: 158708:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1919 previous similar messages [ 5332.605514] Lustre: 158708:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5332.610668] Lustre: 158708:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1919 previous similar messages [ 5341.205384] Lustre: DEBUG MARKER: == sanity-lfsck test 38: LFSCK does not break foreign file and reverse is also true ========================================================== 11:25:14 (1783956314) [ 5354.117612] Lustre: DEBUG MARKER: == sanity-lfsck test 39: LFSCK does not break foreign dir and reverse is also true ========================================================== 11:25:27 (1783956327) [ 5368.260962] Lustre: DEBUG MARKER: == sanity-lfsck test 40a: LFSCK correctly fixes lmm_oi in composite layout ========================================================== 11:25:41 (1783956341) [ 5382.295869] Lustre: DEBUG MARKER: == sanity-lfsck test 41: SEL support in LFSCK ============ 11:25:55 (1783956355) [ 5401.332989] Lustre: DEBUG MARKER: == sanity-lfsck test 42: LFSCK repairs inconsistent MDT-object/OST-object encryption flags ========================================================== 11:26:14 (1783956374) [ 5435.183635] Lustre: *** cfs_fail_loc=1632, val=0*** [ 5447.271284] Lustre: DEBUG MARKER: == sanity-lfsck test 43: LFSCK does not loop endlessly on iget failure in scanning-phase1 ========================================================== 11:27:00 (1783956420) [ 5449.377128] Lustre: Failing over lustre-MDT0001 [ 5449.392111] LustreError: 106510:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8c5506573480 x1870612041331968/t0(0) o103->lustre-MDT0000-osp-MDT0001@0@lo:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'jbd2/dm-1-8.0' uid:0 gid:0 projid:4294967295 [ 5450.319506] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.17@tcp (stopping) [ 5450.723786] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5454.995731] Lustre: server umount lustre-MDT0001 complete [ 5460.594827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.868709] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5460.870027] Lustre: lustre-MDT0001: Aborting client recovery [ 5460.879534] LustreError: 165513:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0001: Aborting recovery [ 5460.882349] LustreError: 165535:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -108 [ 5460.889050] Lustre: 165537:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5460.913074] Lustre: 165537:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 45e93f1d-874e-48e6-bf5a-bb5e2f2866ee@ [ 5460.920482] Lustre: lustre-MDT0001: disconnecting 2 stale clients [ 5460.925927] Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] [ 5460.934134] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] [ 5460.967718] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5460.968168] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:161) [ 5464.102092] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5466.086845] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5466.093065] Lustre: Skipped 2 previous similar messages [ 5466.100572] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5466.946299] Lustre: *** cfs_fail_loc=1e2, val=483*** [ 5470.614359] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid [ 7055.049793] Lustre: DEBUG MARKER: rpc test_43: @@@@@@ FAIL: can't put import for osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid into FULL state after 1475 sec, have [ 7056.855386] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing check_logdir /tmp/test_logs/1783956444 [ 7060.340838] Lustre: DEBUG MARKER: sanity-lfsck test_43: @@@@@@ FAIL: osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid: import is not in expected state [ 7068.383212] Lustre: DEBUG MARKER: == sanity-lfsck test 44: umount while lfsck is stopping == 11:54:01 (1783958041) [ 7069.984530] Lustre: 158709:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 7069.998031] Lustre: 158709:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 188 previous similar messages [ 7070.011661] Lustre: 158709:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 7070.022065] Lustre: 158709:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 188 previous similar messages [ 7070.031641] Lustre: 158709:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 7070.041491] Lustre: 158709:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 188 previous similar messages [ 7070.048619] Lustre: 158709:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 7070.062933] Lustre: 158709:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 188 previous similar messages [ 7070.076054] Lustre: 158709:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 7070.084175] Lustre: 158709:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 188 previous similar messages [ 7070.093502] Lustre: 158709:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 7070.102697] Lustre: 158709:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 188 previous similar messages [ 7077.495336] Lustre: *** cfs_fail_loc=1600, val=3*** [ 7080.230177] Lustre: Failing over lustre-MDT0000 [ 7080.443420] Lustre: server umount lustre-MDT0000 complete [ 7081.956135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7081.964115] LustreError: 160228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7081.964403] Lustre: Skipped 5 previous similar messages [ 7081.987551] LustreError: 160228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 7089.422832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.547877] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7089.823300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7089.828178] Lustre: Skipped 5 previous similar messages [ 7089.869470] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7089.874187] Lustre: Skipped 2 previous similar messages [ 7093.836125] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7094.103680] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7095.275045] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7095.286906] Lustre: Skipped 2 previous similar messages [ 7095.303725] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 7095.350736] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 7095.352614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:621 to 0x280000401:641) [ 7103.069440] Lustre: DEBUG MARKER: == sanity-lfsck test 45: LFSCK should fix UID/GID/PROJID of OST object ========================================================== 11:54:36 (1783958076) [ 7131.557913] Lustre: Failing over lustre-OST0000 [ 7131.623598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7131.632663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7131.643311] Lustre: Skipped 2 previous similar messages [ 7131.728024] Lustre: server umount lustre-OST0000 complete [ 7138.787473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7146.471120] LustreError: 161034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7146.486730] LustreError: 161034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 7149.192558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7149.326966] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 7150.146547] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7150.466058] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7150.470626] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7150.481480] Lustre: Skipped 3 previous similar messages [ 7154.491197] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7159.408303] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7213.401497] Lustre: DEBUG MARKER: rpc test_45: @@@@@@ FAIL: can't put import for os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid into FULL state after 50 sec, have [ 7215.491759] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing check_logdir /tmp/test_logs/1783958132 [ 7218.682716] Lustre: DEBUG MARKER: sanity-lfsck test_45: @@@@@@ FAIL: mds1: import is not in FULL state after 50