[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495206386 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] BRK [0x71a01000, 0x71a01fff] PGTABLE [ 0.000000] BRK [0x71a02000, 0x71a02fff] PGTABLE [ 0.000000] BRK [0x71a03000, 0x71a03fff] PGTABLE [ 0.000000] BRK [0x71a04000, 0x71a04fff] PGTABLE [ 0.000000] BRK [0x71a05000, 0x71a05fff] PGTABLE [ 0.000000] BRK [0x71a06000, 0x71a06fff] PGTABLE [ 0.000000] BRK [0x71a07000, 0x71a07fff] PGTABLE [ 0.000000] BRK [0x71a08000, 0x71a08fff] PGTABLE [ 0.000000] BRK [0x71a09000, 0x71a09fff] PGTABLE [ 0.000000] BRK [0x71a0a000, 0x71a0afff] PGTABLE [ 0.000000] BRK [0x71a0b000, 0x71a0bfff] PGTABLE [ 0.000000] BRK [0x71a0c000, 0x71a0cfff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] On node 0 totalpages: 1076588 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 158 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:63 [ 0.000000] Normal zone: 4536 pages used for memmap [ 0.000000] Normal zone: 290304 pages, LIFO batch:63 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] pcpu-alloc: s221184 r8192 d28672 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001000] hpet clockevent registered [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003182] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008750] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011014] pid_max: default: 32768 minimum: 301 [ 0.013055] LSM: Security Framework initializing [ 0.014052] Yama: becoming mindful. [ 0.015028] SELinux: Initializing. [ 0.016072] *** VALIDATE selinux *** [ 0.024715] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028467] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031102] *** VALIDATE tmpfs *** [ 0.033113] *** VALIDATE proc *** [ 0.034241] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.036261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037173] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043458] debug: unmapping init [mem 0xffffffff9c659000-0xffffffff9c660fff] [ 0.045187] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046688] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047030] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054308] rcu: Hierarchical SRCU implementation. [ 0.056407] smp: Bringing up secondary CPUs ... [ 0.057623] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.061215] smp: Brought up 1 node, 4 CPUs [ 0.063019] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.228100] node 0 deferred pages initialised in 76ms [ 0.230126] devtmpfs: initialized [ 0.231268] x86/mm: Memory block size: 128MB [ 0.236297] gcov: version magic: 0x41383552 [ 0.239319] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.243077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246295] pinctrl core: initialized pinctrl subsystem [ 0.250009] ************************************************************* [ 0.253012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.257013] ** ** [ 0.259011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.262011] ** ** [ 0.265014] ** This means that this kernel is built to expose internal ** [ 0.267009] ** IOMMU data structures, which may compromise security on ** [ 0.270017] ** your system. ** [ 0.272011] ** ** [ 0.276011] ** If you see this message and you are not debugging the ** [ 0.279010] ** kernel, report this immediately to your vendor! ** [ 0.281010] ** ** [ 0.282066] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.284011] ************************************************************* [ 0.286730] NET: Registered protocol family 16 [ 0.289299] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.292067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.294057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.297096] cpuidle: using governor menu [ 0.298876] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.299477] PCI: Using configuration type 1 for base access [ 0.300136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.307118] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.310030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.314078] cryptd: max_cpu_qlen set to 1000 [ 0.317229] ACPI: Added _OSI(Module Device) [ 0.318012] ACPI: Added _OSI(Processor Device) [ 0.320024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.322013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.326585] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.334558] ACPI: Interpreter enabled [ 0.336064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.338014] ACPI: Using IOAPIC for interrupt routing [ 0.340114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.344497] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.355288] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.357038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.360018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.363085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.368728] acpiphp: Slot [2] registered [ 0.370196] acpiphp: Slot [5] registered [ 0.371141] acpiphp: Slot [6] registered [ 0.373193] acpiphp: Slot [7] registered [ 0.375112] acpiphp: Slot [8] registered [ 0.377210] acpiphp: Slot [9] registered [ 0.378122] acpiphp: Slot [10] registered [ 0.380136] acpiphp: Slot [3] registered [ 0.382134] acpiphp: Slot [4] registered [ 0.383095] acpiphp: Slot [11] registered [ 0.385144] acpiphp: Slot [12] registered [ 0.387109] acpiphp: Slot [13] registered [ 0.390095] acpiphp: Slot [14] registered [ 0.391109] acpiphp: Slot [15] registered [ 0.393099] acpiphp: Slot [16] registered [ 0.394078] acpiphp: Slot [17] registered [ 0.396106] acpiphp: Slot [18] registered [ 0.397114] acpiphp: Slot [19] registered [ 0.399091] acpiphp: Slot [20] registered [ 0.400180] acpiphp: Slot [21] registered [ 0.402081] acpiphp: Slot [22] registered [ 0.403087] acpiphp: Slot [23] registered [ 0.404094] acpiphp: Slot [24] registered [ 0.405072] acpiphp: Slot [25] registered [ 0.407088] acpiphp: Slot [26] registered [ 0.408070] acpiphp: Slot [27] registered [ 0.409096] acpiphp: Slot [28] registered [ 0.411094] acpiphp: Slot [29] registered [ 0.412088] acpiphp: Slot [30] registered [ 0.414156] acpiphp: Slot [31] registered [ 0.415074] PCI host bridge to bus 0000:00 [ 0.416015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.418019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.419013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.422024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.424020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.427024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.429151] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.432077] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.435413] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.444023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.447043] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.449020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.451019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.453019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.455504] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.459065] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.461025] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.462722] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.466016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.478014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.483012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.487760] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.492023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.497016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.509016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.514000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.523015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.532017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.554021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.567170] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.575021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.582018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.598020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.607748] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.612014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.618017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.637024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.649850] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.656017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.665015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.677018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.685530] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.692016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.702017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.718016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.727183] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.729391] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.732454] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.736412] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.738216] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.742257] iommu: Default domain type: Passthrough [ 0.744374] SCSI subsystem initialized [ 0.746213] ACPI: bus type USB registered [ 0.748085] usbcore: registered new interface driver usbfs [ 0.750085] usbcore: registered new interface driver hub [ 0.751070] usbcore: registered new device driver usb [ 0.753170] pps_core: LinuxPPS API ver. 1 registered [ 0.755011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.758059] PTP clock support registered [ 0.761064] EDAC MC: Ver: 3.0.0 [ 0.763151] PCI: Using ACPI for IRQ routing [ 0.765015] PCI: pci_cache_line_size set to 64 bytes [ 0.765322] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.765335] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.765339] e820: reserve RAM buffer [mem 0x146e00000-0x147ffffff] [ 0.766000] NetLabel: Initializing [ 0.767010] NetLabel: domain hash size = 128 [ 0.768012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.771084] NetLabel: unlabeled traffic allowed by default [ 0.773193] vgaarb: loaded [ 0.774240] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.776013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.785000] clocksource: Switched to clocksource kvm-clock [ 0.885371] VFS: Disk quotas dquot_6.6.0 [ 0.886567] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.888524] *** VALIDATE ramfs *** [ 0.889688] *** VALIDATE hugetlbfs *** [ 0.892066] pnp: PnP ACPI init [ 0.893553] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.893620] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.893646] pnp 00:02: [dma 2] [ 0.893687] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.893766] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.893827] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.893902] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.894511] pnp: PnP ACPI: found 6 devices [ 0.909962] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.912934] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.914889] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.917210] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.919585] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.921915] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.924575] NET: Registered protocol family 2 [ 0.926568] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.931711] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.935255] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.940996] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.944047] TCP: Hash tables configured (established 65536 bind 65536) [ 0.947394] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.950056] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951946] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.953946] NET: Registered protocol family 1 [ 0.956563] RPC: Registered named UNIX socket transport module. [ 0.958419] RPC: Registered udp transport module. [ 0.959937] RPC: Registered tcp transport module. [ 0.961398] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.963531] NET: Registered protocol family 44 [ 0.965103] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.967318] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.969315] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.971602] PCI: CLS 0 bytes, default 64 [ 0.973073] Unpacking initramfs... [ 2.383371] debug: unmapping init [mem 0xffff8d353cc54000-0xffff8d353ffbffff] [ 2.386927] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.388537] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.390742] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.896736] Initialise system trusted keyrings [ 2.898127] Key type blacklist registered [ 2.899626] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.909331] zbud: loaded [ 2.912265] *** VALIDATE nfs *** [ 2.913410] *** VALIDATE nfs4 *** [ 2.915030] pstore: using deflate compression [ 2.918364] Platform Keyring initialized [ 3.025219] NET: Registered protocol family 38 [ 3.026993] Key type asymmetric registered [ 3.028556] Asymmetric key parser 'x509' registered [ 3.030622] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.034263] io scheduler mq-deadline registered [ 3.035696] io scheduler kyber registered [ 3.037293] io scheduler bfq registered [ 3.040088] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.043484] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.046158] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.048911] ACPI: Power Button [PWRF] [ 3.054232] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.060705] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.073137] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.107211] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.149127] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.176787] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.203331] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.207222] Non-volatile memory driver v1.3 [ 3.208641] Linux agpgart interface v0.103 [ 3.238086] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.240894] vda: detected capacity change from 0 to 74645504 [ 3.306279] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.309126] vdb: detected capacity change from 0 to 1073741824 [ 3.328106] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.329927] vdc: detected capacity change from 0 to 2621440000 [ 3.350538] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.352769] vdd: detected capacity change from 0 to 2621440000 [ 3.378449] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.381277] vde: detected capacity change from 0 to 4294967296 [ 3.409268] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.412290] vdf: detected capacity change from 0 to 4294967296 [ 3.425603] libphy: Fixed MDIO Bus: probed [ 3.441410] usbcore: registered new interface driver usbserial_generic [ 3.444469] usbserial: USB Serial support registered for generic [ 3.447578] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.453317] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.455550] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.458085] mousedev: PS/2 mouse device common for all mice [ 3.461416] rtc_cmos 00:05: RTC can wake from S4 [ 3.464392] rtc_cmos 00:05: registered as rtc0 [ 3.466415] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.469501] intel_pstate: CPU model not supported [ 3.473066] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.477519] hid: raw HID events driver (C) Jiri Kosina [ 3.481050] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.481101] usbcore: registered new interface driver usbhid [ 3.486231] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.486605] usbhid: USB HID core driver [ 3.486719] drop_monitor: Initializing network drop monitor service [ 3.494476] Initializing XFRM netlink socket [ 3.496138] NET: Registered protocol family 10 [ 3.498333] Segment Routing with IPv6 [ 3.499373] NET: Registered protocol family 17 [ 3.500774] mpls_gso: MPLS GSO support [ 3.501620] start plist test [ 3.503857] end plist test [ 3.505560] RAS: Correctable Errors collector initialized. [ 3.507402] AVX version of gcm_enc/dec engaged. [ 3.509063] AES CTR mode by8 optimization enabled [ 3.569712] sched_clock: Marking stable (3569633901, 0)->(4483416644, -913782743) [ 3.573245] registered taskstats version 1 [ 3.575507] Loading compiled-in X.509 certificates [ 3.577688] zswap: loaded using pool lzo/zbud [ 3.597742] Key type big_key registered [ 3.608715] Key type encrypted registered [ 3.610462] ima: No TPM chip found, activating TPM-bypass! [ 3.612591] ima: Allocated hash algorithm: sha1 [ 3.614783] ima: No architecture policies found [ 3.616842] evm: Initialising EVM extended attributes: [ 3.618627] evm: security.selinux [ 3.619819] evm: security.ima [ 3.620939] evm: security.capability [ 3.622143] evm: HMAC attrs: 0x1 [ 3.624285] rtc_cmos 00:05: setting system clock to 2026-07-14 16:04:39 UTC (1784045079) [ 3.629770] debug: unmapping init [mem 0xffffffff9d603000-0xffffffff9d7fffff] [ 3.632975] debug: unmapping init [mem 0xffffffff9c382000-0xffffffff9c658fff] [ 3.642057] Write protecting the kernel read-only data: 28672k [ 3.644740] debug: unmapping init [mem 0xffffffff9aa03000-0xffffffff9abfffff] [ 3.647747] debug: unmapping init [mem 0xffffffff9b314000-0xffffffff9b3fffff] [ 3.682753] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.691079] systemd[1]: Detected virtualization kvm. [ 3.693248] systemd[1]: Detected architecture x86-64. [ 3.695533] systemd[1]: Running in initial RAM disk. [ 3.724298] systemd[1]: No hostname configured. [ 3.726272] systemd[1]: Set hostname to . [ 3.728260] random: systemd: uninitialized urandom read (16 bytes read) [ 3.730880] systemd[1]: Initializing machine ID from random generator. [ 3.862097] random: systemd: uninitialized urandom read (16 bytes read) [ 3.864809] systemd[1]: Listening on Journal Socket. [ 3.869180] random: systemd: uninitialized urandom read (16 bytes read) [ 3.876502] systemd[1]: Started Memstrack Anylazing Service. [ 3.890608] systemd[1]: Starting Apply Kernel Variables... [ 4.504303] device-mapper: uevent: version 1.0.3 [ 4.505940] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 5.218147] random: fast init done [ 5.231305] libata version 3.00 loaded. [ 5.234233] ata_piix 0000:00:01.1: version 2.13 [ 5.278653] virtio_net virtio0 ens2: renamed from eth0 [ 5.328193] scsi host0: ata_piix [ 5.339407] scsi host1: ata_piix [ 5.341080] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.343729] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.126373] random: crng init done [ 10.127651] random: 7 urandom warning(s) missed due to ratelimiting [ 10.512904] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 11.573845] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.815119] SELinux: Disabled at runtime. [ 11.877652] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.886324] systemd[1]: Detected virtualization kvm. [ 11.888229] systemd[1]: Detected architecture x86-64. [ 12.365252] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.368706] systemd[1]: Stopped Switch Root. [ 12.373814] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.381340] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.385385] systemd[1]: Stopped Journal Service. [ 12.392195] systemd[1]: Starting Journal Service... [ 12.397107] systemd[1]: Listening on udev Control Socket. [ 12.658211] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 12.891114] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 13.197979] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.228364] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.342229] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.352728] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 13.352753] EDAC sbridge: Ver: 1.1.2 [ 14.719907] Key type dns_resolver registered [ 15.032690] NFS: Registering the id_resolver key type [ 15.035348] Key type id_resolver registered [ 15.037580] Key type id_legacy registered [ 42.223348] libcfs: loading out-of-tree module taints kernel. [ 42.244191] Key type ._llcrypt registered [ 42.245986] Key type .llcrypt registered [ 42.296547] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 57.027440] hrtimer: interrupt took 25429464 ns [ 59.749464] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 63.035247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.065504] alg: No test for adler32 (adler32-zlib) [ 65.331747] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 67.112106] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 68.879260] Key type lgssc registered [ 71.957146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 92.968421] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 137.244640] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 152.314436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 152.355051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.646684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 153.703660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 153.862097] Lustre: lustre-MDT0000: new disk, initializing [ 154.019856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 154.056651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.504577] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.659609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.816461] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 176.884452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 176.894767] Lustre: Skipped 1 previous similar message [ 177.038261] Lustre: lustre-MDT0001: new disk, initializing [ 177.114490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.147742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 177.164771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.746557] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.272856] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.413338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.687269] Lustre: lustre-OST0000: new disk, initializing [ 199.707125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.722843] Lustre: 8443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.853559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.844796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.870756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.094434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.881764] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.386171] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.528475] Lustre: lustre-OST0001: new disk, initializing [ 222.530297] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 222.533421] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.588320] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.418424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.430071] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 228.710786] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 231.358708] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.579300] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.899143] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 259.109909] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.494579] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing yml_node [ 272.061877] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 275.066780] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 278.341398] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 280.415519] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Tue Jul 14 12:09:14 EDT 2026 [ 297.242346] Lustre: DEBUG MARKER: excepting tests: 18b 23b [ 308.056496] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 315.883671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.911502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.931524] Lustre: Skipped 2 previous similar messages [ 320.480903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 320.486438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.515160] Lustre: Skipped 2 previous similar messages [ 320.544799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.281593] Lustre: server umount lustre-MDT0000 complete [ 330.730050] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045406 with bad export cookie 15229039216191913689 [ 330.737851] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 330.747370] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 331.090410] Lustre: server umount lustre-MDT0001 complete [ 348.063179] Lustre: 3639:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784045407/real 1784045407] req@ffff8d35bf421c00 x1870706918923264/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784045423 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 348.097302] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.280053] Lustre: server umount lustre-OST0000 complete [ 351.534602] Lustre: 3637:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784045411/real 1784045411] req@ffff8d348347df80 x1870706918923520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784045427 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 352.735098] Lustre: 3640:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784045412/real 1784045412] req@ffff8d34832ca680 x1870706918923776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784045428 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 356.256104] Lustre: 3639:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784045416/real 1784045416] req@ffff8d34832cb800 x1870706918924160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784045432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 358.357480] Lustre: server umount lustre-OST0001 complete [ 375.261950] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 377.922059] Key type lgssc unregistered [ 378.232549] LNet: 14787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.242311] LNetError: 14787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 378.252613] LNet: Removed LNI 192.168.202.155@tcp [ 379.089853] Key type .llcrypt unregistered [ 379.095724] Key type ._llcrypt unregistered [ 402.308769] Key type ._llcrypt registered [ 402.314471] Key type .llcrypt registered [ 402.404522] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 416.887285] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 418.304555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 418.326316] alg: No test for adler32 (adler32-zlib) [ 419.410353] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 419.719594] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 421.495177] Key type lgssc registered [ 422.604387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 476.192520] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 488.904908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 488.970930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.178801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 490.204515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 490.267394] Lustre: lustre-MDT0000: new disk, initializing [ 490.325751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.343664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 495.302398] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 509.395803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.505390] Lustre: 19243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 509.526953] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 509.532193] Lustre: Skipped 1 previous similar message [ 509.597861] Lustre: lustre-MDT0001: new disk, initializing [ 509.655939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 509.684880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 509.698840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 513.992201] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 518.901763] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 530.492448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 530.809760] Lustre: lustre-OST0000: new disk, initializing [ 530.814870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 530.822338] Lustre: 21181:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 530.894731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 534.169650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 534.179740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 534.314815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 537.962709] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 552.542448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 552.689713] Lustre: lustre-OST0001: new disk, initializing [ 552.698727] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 552.705842] Lustre: 22205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 552.791338] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 559.891912] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 562.223450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 562.237906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 562.288842] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 572.116549] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 579.804260] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 589.166651] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 12:14:23 (1784045663) === [ 592.374648] Lustre: DEBUG MARKER: == sanity-lfsck test 0: Control LFSCK manually =========== 12:14:26 (1784045666) [ 592.771355] Lustre: 19249:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 592.787447] Lustre: 19249:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 592.796223] Lustre: 19249:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 592.803652] Lustre: 19249:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 592.818223] Lustre: 19249:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 592.825665] Lustre: 19249:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 593.330052] Lustre: 20504:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 593.346159] Lustre: 20504:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 8 previous similar messages [ 593.354821] Lustre: 20504:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 593.362549] Lustre: 20504:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 593.371000] Lustre: 20504:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 593.379025] Lustre: 20504:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 593.384319] Lustre: 20504:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 593.393574] Lustre: 20504:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 593.404396] Lustre: 20504:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 593.411993] Lustre: 20504:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 593.420582] Lustre: 20504:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 593.426353] Lustre: 20504:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 594.388286] Lustre: 19248:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 594.395967] Lustre: 19248:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 38 previous similar messages [ 594.403888] Lustre: 19248:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 594.417236] Lustre: 19248:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 594.434620] Lustre: 19248:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 594.444957] Lustre: 19248:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 594.461665] Lustre: 19248:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 594.471486] Lustre: 19248:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 594.484495] Lustre: 19248:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 594.495390] Lustre: 19248:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 594.508126] Lustre: 19248:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 594.518569] Lustre: 19248:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 596.388814] Lustre: 19249:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 596.398560] Lustre: 19249:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 104 previous similar messages [ 596.404727] Lustre: 19249:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 596.409720] Lustre: 19249:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 105 previous similar messages [ 596.493617] Lustre: 19250:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 596.500179] Lustre: 19250:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 107 previous similar messages [ 596.505381] Lustre: 19250:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 596.511050] Lustre: 19250:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 107 previous similar messages [ 596.518353] Lustre: 19250:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 596.527881] Lustre: 19250:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 107 previous similar messages [ 596.537981] Lustre: 19250:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 596.542751] Lustre: 19250:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 107 previous similar messages [ 600.914391] Lustre: *** cfs_fail_loc=1600, val=3*** [ 602.891186] Lustre: 23392:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 274, rollback = 2 [ 602.899227] Lustre: 23392:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 123 previous similar messages [ 602.901716] Lustre: 23576:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 602.905718] Lustre: 23392:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 602.905731] Lustre: 23392:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 120 previous similar messages [ 602.905738] Lustre: 23392:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 602.905741] Lustre: 23392:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 120 previous similar messages [ 602.905748] Lustre: 23392:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 602.905752] Lustre: 23392:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 120 previous similar messages [ 602.905758] Lustre: 23392:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 602.905763] Lustre: 23392:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 120 previous similar messages [ 602.993307] Lustre: 23576:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 124 previous similar messages [ 605.530983] Lustre: *** cfs_fail_loc=1600, val=3*** [ 618.023001] Lustre: 23586:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 274, rollback = 2 [ 618.032933] Lustre: 23592:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 618.044613] Lustre: 23586:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 57 previous similar messages [ 618.044642] Lustre: 23586:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 618.044646] Lustre: 23586:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 57 previous similar messages [ 618.044651] Lustre: 23586:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 618.044653] Lustre: 23586:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 57 previous similar messages [ 618.044658] Lustre: 23586:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 618.044661] Lustre: 23586:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 57 previous similar messages [ 618.044666] Lustre: 23586:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 618.044669] Lustre: 23586:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 57 previous similar messages [ 618.176942] Lustre: 23592:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 61 previous similar messages [ 622.560892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 622.573435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.600680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.591123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.595560] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.608623] Lustre: Skipped 2 previous similar messages [ 628.809878] Lustre: server umount lustre-MDT0000 complete [ 633.260129] LustreError: 19232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045709 with bad export cookie 15014200346482819066 [ 633.268157] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.291051] LustreError: 19232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 633.685681] Lustre: server umount lustre-MDT0001 complete [ 648.204698] Lustre: server umount lustre-OST0000 complete [ 649.186821] Lustre: 16405:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784045709/real 1784045709] req@ffff8d348347ca80 x1870707289079040/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784045725 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 649.219261] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 652.298546] Lustre: server umount lustre-OST0001 complete [ 660.436237] Lustre: DEBUG MARKER: == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 12:15:35 (1784045735) [ 681.718388] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 694.828316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.169853] LustreError: 26264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 695.187662] LustreError: 26264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 695.257763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 700.384924] LustreError: 26265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.798712] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 705.509703] LustreError: 26264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 709.367594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.812052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 714.149558] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 717.800044] Lustre: 27404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 725.789459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 726.308422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 734.510678] LustreError: 27759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 734.680142] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 739.809360] LustreError: 27756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 740.880304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 745.866118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 751.075785] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:40 to 0x2c0000401:65) [ 752.524925] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 760.876873] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 764.848280] Lustre: 29273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 766.566080] Lustre: 26261:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 766.579077] Lustre: 26261:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 10 previous similar messages [ 766.586167] Lustre: 26261:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 766.597964] Lustre: 26261:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 766.610799] Lustre: 26261:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 766.617806] Lustre: 26261:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 766.625705] Lustre: 26261:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 766.631181] Lustre: 26261:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 766.637169] Lustre: 26261:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 766.645229] Lustre: 26261:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 766.649825] Lustre: 26261:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 766.656682] Lustre: 26261:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 772.048607] Lustre: *** cfs_fail_loc=1501, val=0*** [ 779.797755] Lustre: Failing over lustre-MDT0000 [ 780.033600] Lustre: server umount lustre-MDT0000 complete [ 781.279651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 781.288892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.312268] LustreError: 26264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 781.330498] LustreError: 26264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 791.372620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 791.546928] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 791.920498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.932591] Lustre: Skipped 1 previous similar message [ 792.024173] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 797.162723] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 797.168101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 797.203319] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 797.284264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 797.289541] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 797.868956] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 801.503794] Lustre: *** cfs_fail_loc=1505, val=0*** [ 810.645263] Lustre: DEBUG MARKER: == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 12:18:04 (1784045884) [ 812.101783] Lustre: 26259:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 812.112797] Lustre: 26259:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 322 previous similar messages [ 812.119896] Lustre: 26259:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 812.125215] Lustre: 26259:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 812.130650] Lustre: 26259:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 812.136350] Lustre: 26259:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 812.143085] Lustre: 26259:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 812.147991] Lustre: 26259:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 812.152982] Lustre: 26259:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 812.157627] Lustre: 26259:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 812.162287] Lustre: 26259:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 812.167308] Lustre: 26259:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 817.057272] Lustre: *** cfs_fail_loc=1502, val=0*** [ 828.341194] Lustre: Failing over lustre-MDT0000 [ 828.573574] Lustre: server umount lustre-MDT0000 complete [ 832.998533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 833.008503] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.017796] LustreError: 26264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 833.017807] LustreError: 26264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 833.102833] Lustre: Skipped 6 previous similar messages [ 840.858419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 841.024643] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 841.184161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 841.196337] Lustre: Skipped 6 previous similar messages [ 841.454785] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 846.285802] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 846.816338] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 846.819298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 846.840631] Lustre: Skipped 3 previous similar messages [ 846.881130] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 846.919206] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:169 to 0x2c0000401:193) [ 846.922497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 850.326626] Lustre: *** cfs_fail_loc=1505, val=0*** [ 859.514768] Lustre: DEBUG MARKER: == sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 12:18:53 (1784045933) [ 866.263798] Lustre: *** cfs_fail_loc=1504, val=0*** [ 866.274620] Lustre: *** cfs_fail_loc=1504, val=0*** [ 866.279260] Lustre: Skipped 1 previous similar message [ 876.774433] Lustre: Failing over lustre-MDT0000 [ 877.054407] Lustre: server umount lustre-MDT0000 complete [ 877.541041] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.556048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 877.563146] LustreError: 26259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 877.563158] LustreError: 26259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 877.563954] Lustre: Skipped 2 previous similar messages [ 889.309961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 889.455260] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 889.696790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.701615] Lustre: Skipped 1 previous similar message [ 889.753497] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 894.856830] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 894.959057] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 894.980917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 894.984792] Lustre: Skipped 3 previous similar messages [ 895.018576] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 895.098131] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:257) [ 895.113221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:233 to 0x280000401:257) [ 898.580428] Lustre: *** cfs_fail_loc=1505, val=0*** [ 907.659198] Lustre: DEBUG MARKER: == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 12:19:41 (1784045981) [ 909.138650] Lustre: 32362:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 909.159324] Lustre: 32362:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 644 previous similar messages [ 909.171503] Lustre: 32362:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 909.179395] Lustre: 32362:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 909.191958] Lustre: 32362:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 909.207402] Lustre: 32362:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 909.219437] Lustre: 32362:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 909.229248] Lustre: 32362:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 909.244624] Lustre: 32362:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 909.267197] Lustre: 32362:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 909.277929] Lustre: 32362:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 909.297231] Lustre: 32362:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 914.811333] Lustre: *** cfs_fail_loc=1603, val=0*** [ 923.323804] Lustre: Failing over lustre-MDT0000 [ 923.743769] Lustre: server umount lustre-MDT0000 complete [ 925.665703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.674933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 925.695470] Lustre: Skipped 2 previous similar messages [ 936.059694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 936.205639] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 936.422597] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 941.235806] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 941.538119] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 941.541221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 941.554643] Lustre: Skipped 3 previous similar messages [ 941.565777] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 941.602439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:297 to 0x280000401:321) [ 941.608559] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:297 to 0x2c0000401:321) [ 953.259672] Lustre: DEBUG MARKER: == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 12:20:27 (1784046027) [ 960.445996] Lustre: *** cfs_fail_loc=1604, val=0*** [ 968.924968] Lustre: Failing over lustre-MDT0000 [ 969.319390] Lustre: server umount lustre-MDT0000 complete [ 972.255837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 972.263765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.266934] LustreError: 32559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.266945] LustreError: 32559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 972.331537] Lustre: Skipped 4 previous similar messages [ 982.045250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 982.164700] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 982.428519] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 987.622884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 987.624452] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 987.636570] Lustre: Skipped 3 previous similar messages [ 987.684762] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 987.723352] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:361 to 0x2c0000401:385) [ 987.726650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:361 to 0x280000401:385) [ 988.050919] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 998.243160] Lustre: DEBUG MARKER: == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 12:21:12 (1784046072) [ 1004.260483] Lustre: *** cfs_fail_loc=1605, val=0*** [ 1012.519612] Lustre: Failing over lustre-MDT0000 [ 1012.836424] Lustre: server umount lustre-MDT0000 complete [ 1022.990858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1023.156473] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.416658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.423369] Lustre: Skipped 2 previous similar messages [ 1023.454335] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1027.497368] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1028.578041] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1028.589465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1028.602180] Lustre: Skipped 3 previous similar messages [ 1028.615671] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1028.656484] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:425 to 0x2c0000401:449) [ 1028.656583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:425 to 0x280000401:449) [ 1038.744655] Lustre: DEBUG MARKER: == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 12:21:52 (1784046112) [ 1040.932605] Lustre: 26259:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1040.950084] Lustre: 26259:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 968 previous similar messages [ 1040.968440] Lustre: 26259:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1041.002210] Lustre: 26259:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1041.020918] Lustre: 26259:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1041.036552] Lustre: 26259:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1041.055750] Lustre: 26259:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1041.067571] Lustre: 26259:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1041.083658] Lustre: 26259:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1041.100772] Lustre: 26259:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1041.110985] Lustre: 26259:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1041.122953] Lustre: 26259:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1047.694394] Lustre: *** cfs_fail_loc=161d, val=0*** [ 1056.412029] Lustre: Failing over lustre-MDT0000 [ 1056.556164] Lustre: server umount lustre-MDT0000 complete [ 1059.295836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1059.295874] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.356273] Lustre: Skipped 8 previous similar messages [ 1067.425673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1067.553182] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1067.930590] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1073.126282] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1073.140585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1073.163359] Lustre: Skipped 3 previous similar messages [ 1073.181615] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1073.243024] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 1073.254580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:489 to 0x280000401:513) [ 1074.174220] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1083.869589] Lustre: DEBUG MARKER: == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 12:22:38 (1784046158) [ 1086.319442] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1097.730917] Lustre: DEBUG MARKER: == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 12:22:52 (1784046172) [ 1104.574770] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1105.710312] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1118.389372] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 12:23:12 (1784046192) [ 1154.188807] Lustre: Failing over lustre-MDT0000 [ 1154.432923] Lustre: server umount lustre-MDT0000 complete [ 1155.044086] LustreError: 28103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1155.049406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1155.061196] LustreError: 28103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 1159.909849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1170.179871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1170.391121] Lustre: 16405:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784046230/real 1784046230] req@ffff8d348ab52300 x1870707289714944/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784046246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1170.419139] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1183.323577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1183.376541] Lustre: lustre-MDT0000: reset Object Index mappings [ 1196.469773] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1200.974833] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1201.639429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1201.652544] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1201.652904] Lustre: Skipped 3 previous similar messages [ 1201.693693] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1201.729729] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:609) [ 1201.734379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:609) [ 1204.654193] LustreError: 42922:0:(lfsck_engine.c:1018:lfsck_master_engine()) lustre-MDT0000-osd: master engine fail to verify the .lustre/lost+found/, go ahead: rc = -115 [ 1204.665148] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1206.753280] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1206.762246] Lustre: Skipped 1 previous similar message [ 1213.854285] Lustre: Failing over lustre-MDT0000 [ 1214.082758] Lustre: server umount lustre-MDT0000 complete [ 1216.995323] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.039222] Lustre: Skipped 7 previous similar messages [ 1227.803152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1233.483487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:641) [ 1233.488232] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:641) [ 1233.591616] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1237.362199] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1247.023442] Lustre: DEBUG MARKER: == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 12:25:21 (1784046321) [ 1249.657342] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1261.640251] Lustre: Failing over lustre-MDT0000 [ 1261.903057] Lustre: server umount lustre-MDT0000 complete [ 1264.117199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1264.125419] LustreError: Skipped 1 previous similar message [ 1267.390587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1276.923693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1279.442498] Lustre: 16402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784046339/real 1784046339] req@ffff8d3590166680 x1870707289816192/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784046355 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1288.549801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1288.570501] Lustre: lustre-MDT0000: reset Object Index mappings [ 1290.015592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.018863] Lustre: Skipped 3 previous similar messages [ 1290.073661] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1290.079547] Lustre: Skipped 1 previous similar message [ 1294.985622] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1295.331064] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1295.334764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1295.349400] Lustre: Skipped 1 previous similar message [ 1295.367404] Lustre: Skipped 7 previous similar messages [ 1295.406363] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1295.418420] Lustre: Skipped 1 previous similar message [ 1295.446107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:705) [ 1295.449588] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:705) [ 1299.095750] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1299.108695] Lustre: Skipped 2 previous similar messages [ 1307.300016] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1307.303468] Lustre: Skipped 7 previous similar messages [ 1312.781973] Lustre: Failing over lustre-MDT0000 [ 1315.063290] Lustre: server umount lustre-MDT0000 complete [ 1326.929042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1327.046491] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1327.054418] LustreError: Skipped 2 previous similar messages [ 1332.818317] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:737) [ 1332.825018] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:737) [ 1334.277431] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1337.992953] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1338.002970] Lustre: Skipped 84 previous similar messages [ 1348.882576] Lustre: DEBUG MARKER: == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 12:27:03 (1784046423) [ 1350.099358] Lustre: 28103:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1350.104953] Lustre: 28103:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1306 previous similar messages [ 1350.109567] Lustre: 28103:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1350.115178] Lustre: 28103:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1306 previous similar messages [ 1350.122216] Lustre: 28103:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1350.127031] Lustre: 28103:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1306 previous similar messages [ 1350.131438] Lustre: 28103:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1350.138251] Lustre: 28103:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1306 previous similar messages [ 1350.150168] Lustre: 28103:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1350.158391] Lustre: 28103:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1306 previous similar messages [ 1350.167749] Lustre: 28103:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1350.173920] Lustre: 28103:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1306 previous similar messages [ 1358.400861] Lustre: *** cfs_fail_loc=1600, val=1*** [ 1379.303731] Lustre: DEBUG MARKER: == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 12:27:33 (1784046453) [ 1391.007371] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1391.013380] Lustre: Skipped 13 previous similar messages [ 1413.329146] Lustre: DEBUG MARKER: == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 12:28:07 (1784046487) [ 1431.579612] Lustre: Failing over lustre-MDT0000 [ 1431.946975] Lustre: server umount lustre-MDT0000 complete [ 1435.103757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1435.111272] LustreError: 26265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.137853] LustreError: 26265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 84 previous similar messages [ 1441.179096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1441.524549] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1441.541888] Lustre: Skipped 1 previous similar message [ 1446.403401] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1446.892178] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1446.901842] Lustre: Skipped 1 previous similar message [ 1446.905101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1446.925144] Lustre: Skipped 7 previous similar messages [ 1446.957704] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1446.977048] Lustre: Skipped 1 previous similar message [ 1447.075161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:854 to 0x280000401:897) [ 1447.077903] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:853 to 0x2c0000401:897) [ 1458.564896] Lustre: DEBUG MARKER: == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 12:28:52 (1784046532) [ 1478.388762] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1501.153186] Lustre: 52943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1526.730749] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1531.044611] Lustre: 54079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1539.248698] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1539.256419] Lustre: Skipped 81 previous similar messages [ 1542.040044] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1543.072315] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1544.095247] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1546.143110] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1546.145846] Lustre: Skipped 1 previous similar message [ 1546.613435] Lustre: Failing over lustre-MDT0000 [ 1546.953305] Lustre: server umount lustre-MDT0000 complete [ 1549.285844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.299062] Lustre: Skipped 13 previous similar messages [ 1556.722673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1562.230550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:947 to 0x280000401:993) [ 1562.232842] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:946 to 0x2c0000401:961) [ 1562.579185] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1573.557586] Lustre: DEBUG MARKER: == sanity-lfsck test 8: LFSCK state machine ============== 12:30:47 (1784046647) [ 1577.451961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1577.458026] Lustre: Skipped 3 previous similar messages [ 1583.956422] Lustre: server umount lustre-MDT0000 complete [ 1587.688520] LustreError: 29275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046663 with bad export cookie 15014200346483033280 [ 1587.697571] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1587.698260] LustreError: 29275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1587.718881] LustreError: Skipped 2 previous similar messages [ 1588.089424] Lustre: server umount lustre-MDT0001 complete [ 1603.185132] Lustre: server umount lustre-OST0000 complete [ 1616.447410] Lustre: server umount lustre-OST0001 complete [ 1623.160418] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 1633.297215] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1684.294677] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1695.567806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.737874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1695.757469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1695.823639] Lustre: lustre-MDT0000: new disk, initializing [ 1695.899530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1701.084805] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1712.525566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1712.614751] Lustre: 59145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1712.638490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1712.643570] Lustre: Skipped 1 previous similar message [ 1712.768210] Lustre: lustre-MDT0001: new disk, initializing [ 1712.873661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1712.886703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1717.389449] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1722.557239] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1728.952858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1729.083771] Lustre: lustre-OST0000: new disk, initializing [ 1729.087512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1729.092706] Lustre: 60776:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1730.293301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1730.299675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1730.412825] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1735.807510] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1747.088349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1747.218455] Lustre: lustre-OST0001: new disk, initializing [ 1747.232636] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1747.256495] Lustre: 61647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1748.903929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1748.912490] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1748.964835] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1753.469984] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1763.874639] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1766.983861] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1780.373251] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1781.231796] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1781.238036] Lustre: Skipped 19 previous similar messages [ 1786.030134] Lustre: *** cfs_fail_loc=1601, val=2*** [ 1786.034478] Lustre: Skipped 16 previous similar messages [ 1803.368233] Lustre: Failing over lustre-MDT0000 [ 1805.303465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1805.312384] LustreError: Skipped 2 previous similar messages [ 1805.685880] Lustre: server umount lustre-MDT0000 complete [ 1817.370554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1817.889560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.891653] Lustre: Skipped 7 previous similar messages [ 1817.921454] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1817.925871] Lustre: Skipped 1 previous similar message [ 1822.621776] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1823.206443] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1823.215282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1823.220177] Lustre: Skipped 1 previous similar message [ 1823.243922] Lustre: Skipped 7 previous similar messages [ 1823.262310] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1823.270733] Lustre: Skipped 1 previous similar message [ 1823.355746] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1823.369626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:65) [ 1823.375162] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:65) [ 1832.674538] Lustre: Failing over lustre-MDT0000 [ 1833.074696] Lustre: server umount lustre-MDT0000 complete [ 1844.286357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1849.892508] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1849.911199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:97) [ 1849.911199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:97) [ 1850.048833] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1855.568725] Lustre: Failing over lustre-MDT0000 [ 1858.035982] Lustre: server umount lustre-MDT0000 complete [ 1867.548752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1872.989922] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1873.441570] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:129) [ 1873.442086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:129) [ 1878.560307] Lustre: *** cfs_fail_loc=1602, val=2*** [ 1892.436600] Lustre: DEBUG MARKER: == sanity-lfsck test 9a: LFSCK speed control (1) ========= 12:36:06 (1784046966) [ 1906.948920] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1924.012114] Lustre: 68622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1946.476423] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1949.699770] Lustre: 69758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1958.474031] Lustre: 59152:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1958.489780] Lustre: 59152:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 2818 previous similar messages [ 1958.500856] Lustre: 59152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1958.512181] Lustre: 59152:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 2818 previous similar messages [ 1958.518877] Lustre: 59152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1958.526368] Lustre: 59152:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 2818 previous similar messages [ 1958.536881] Lustre: 59152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1958.549939] Lustre: 59152:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 2818 previous similar messages [ 1958.556927] Lustre: 59152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1958.562908] Lustre: 59152:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 2818 previous similar messages [ 1958.574395] Lustre: 59152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1958.580809] Lustre: 59152:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 2818 previous similar messages [ 2060.252534] Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) ========= 12:38:54 (1784047134) [ 2116.194467] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2116.206325] Lustre: Skipped 4 previous similar messages [ 2139.538312] Lustre: *** cfs_fail_loc=160c, val=0*** [ 2139.541273] Lustre: Skipped 7 previous similar messages [ 2176.628457] Lustre: DEBUG MARKER: == sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 12:40:50 (1784047250) [ 2225.631905] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2226.646408] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2226.650611] Lustre: Skipped 50 previous similar messages [ 2228.648411] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2228.651876] Lustre: Skipped 134 previous similar messages [ 2232.687777] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2232.689214] Lustre: Skipped 221 previous similar messages [ 2240.735098] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2240.738790] Lustre: Skipped 363 previous similar messages [ 2256.738817] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2256.743094] Lustre: Skipped 798 previous similar messages [ 2288.741334] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2288.745284] Lustre: Skipped 1575 previous similar messages [ 2297.165365] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2297.170805] Lustre: Skipped 2599 previous similar messages [ 2547.150876] Lustre: DEBUG MARKER: == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 12:47:01 (1784047621) [ 2711.937530] Lustre: 63331:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 2711.951234] Lustre: 63331:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 36831 previous similar messages [ 2711.964033] Lustre: 63331:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 2711.970071] Lustre: 63331:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 36832 previous similar messages [ 2711.976295] Lustre: 63331:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 2711.982122] Lustre: 63331:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 36832 previous similar messages [ 2711.989755] Lustre: 63331:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 2711.995313] Lustre: 63331:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 36832 previous similar messages [ 2712.001913] Lustre: 63331:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 2712.007129] Lustre: 63331:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 36832 previous similar messages [ 2712.012321] Lustre: 63331:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 2712.016973] Lustre: 63331:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 36832 previous similar messages [ 2718.177431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.179904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.191234] Lustre: Skipped 20 previous similar messages [ 2718.195631] Lustre: Skipped 1 previous similar message [ 2718.209199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2718.213529] LustreError: Skipped 1 previous similar message [ 2723.297886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2723.304414] Lustre: Skipped 5 previous similar messages [ 2724.074716] Lustre: server umount lustre-MDT0000 complete [ 2727.950229] LustreError: 59136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047803 with bad export cookie 15014200346483052383 [ 2727.955663] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2727.959483] LustreError: 59136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2727.977214] LustreError: Skipped 3 previous similar messages [ 2728.299352] Lustre: server umount lustre-MDT0001 complete [ 2742.139470] Lustre: server umount lustre-OST0000 complete [ 2744.799144] Lustre: 16402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784047804/real 1784047804] req@ffff8d35b6e7d500 x1870707293974400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784047820 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2746.027972] Lustre: server umount lustre-OST0001 complete [ 2752.777937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 2761.139502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2776.736235] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2781.856093] LustreError: 75187:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.155@tcp: failed processing log, type 4: rc = -110 [ 2807.583222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2807.595701] Lustre: Skipped 2 previous similar messages [ 2815.393463] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2819.392811] Lustre: 75771:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2819.416055] Lustre: *** cfs_fail_loc=160e, val=3*** [ 2822.436109] Lustre: 75771:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2832.204328] Lustre: DEBUG MARKER: == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 12:51:46 (1784047906) [ 2847.034704] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 2862.845507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.349791] LustreError: 75212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.385690] LustreError: 75212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 52 previous similar messages [ 2863.543831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3272 to 0x280000401:3297) [ 2869.105367] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2878.831839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2884.542045] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2888.423151] Lustre: 78438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2905.254418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2910.693672] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3207 to 0x2c0000401:3233) [ 2912.344712] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2920.314156] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2924.075459] Lustre: 79934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2929.134324] Lustre: *** cfs_fail_loc=160d, val=0*** [ 2929.661368] Lustre: *** cfs_fail_loc=160d, val=0*** [ 2929.663379] Lustre: Skipped 3 previous similar messages [ 2936.994665] Lustre: Failing over lustre-OST0000 [ 2937.114205] Lustre: server umount lustre-OST0000 complete [ 2940.389486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.415984] Lustre: Skipped 2 previous similar messages [ 2940.418962] LustreError: 80096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.444184] LustreError: 80096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [ 2946.235806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2946.519427] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2946.543238] Lustre: Skipped 2 previous similar messages [ 2948.386247] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2948.399445] Lustre: Skipped 2 previous similar messages [ 2948.435486] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2948.435933] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2948.438446] Lustre: *** cfs_fail_loc=215, val=0*** [ 2948.449360] Lustre: Skipped 2 previous similar messages [ 2948.486703] Lustre: Skipped 11 previous similar messages [ 2953.302668] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2953.696240] Lustre: *** cfs_fail_loc=215, val=0*** [ 2953.697586] Lustre: Skipped 3 previous similar messages [ 2957.062778] Lustre: 81332:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2957.084721] Lustre: 81332:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2958.822645] Lustre: *** cfs_fail_loc=215, val=0*** [ 2958.826148] Lustre: Skipped 2 previous similar messages [ 2959.950160] Lustre: Failing over lustre-OST0000 [ 2960.111367] Lustre: server umount lustre-OST0000 complete [ 2968.329633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2969.594519] Lustre: *** cfs_fail_loc=215, val=0*** [ 2969.606554] Lustre: Skipped 1 previous similar message [ 2974.119470] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2974.687908] Lustre: *** cfs_fail_loc=215, val=0*** [ 2974.692456] Lustre: Skipped 1 previous similar message [ 2981.856769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2983.904191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2983.922467] Lustre: Skipped 3 previous similar messages [ 2985.888652] Lustre: server umount lustre-MDT0000 complete [ 2989.961900] LustreError: 75194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048065 with bad export cookie 15014200346484616911 [ 2989.970034] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2989.974359] LustreError: 75194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2990.248372] Lustre: server umount lustre-MDT0001 complete [ 3005.309272] Lustre: server umount lustre-OST0000 complete [ 3020.085990] Lustre: server umount lustre-OST0001 complete [ 3029.255715] Lustre: DEBUG MARKER: == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 12:55:03 (1784048103) [ 3045.146978] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 3057.259871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.369856] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3072.480556] LustreError: 84579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3072.505679] LustreError: 84579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3074.452659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.516463] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3082.900248] Lustre: 85721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3090.524355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3099.125311] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3103.154873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3362 to 0x280000401:3393) [ 3108.556258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.963985] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3207 to 0x2c0000401:3265) [ 3115.360966] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3122.918345] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3126.872815] Lustre: 87589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3164.134941] Lustre: DEBUG MARKER: == sanity-lfsck test 12b: auto detect Lustre device ====== 12:57:17 (1784048237) [ 3183.523585] Lustre: DEBUG MARKER: == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 12:57:37 (1784048257) [ 3184.893159] Lustre: *** cfs_fail_loc=160f, val=0*** [ 3193.171979] Lustre: DEBUG MARKER: == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 12:57:47 (1784048267) [ 3196.309523] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3196.311539] Lustre: Skipped 3 previous similar messages [ 3247.104219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.109147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.121104] Lustre: Skipped 9 previous similar messages [ 3247.131589] Lustre: Skipped 3 previous similar messages [ 3250.804695] Lustre: server umount lustre-MDT0000 complete [ 3254.688126] LustreError: 84560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048330 with bad export cookie 15014200346484625367 [ 3254.688930] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3254.705062] LustreError: 84560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3255.170528] Lustre: server umount lustre-MDT0001 complete [ 3268.379944] Lustre: server umount lustre-OST0000 complete [ 3281.402044] Lustre: server umount lustre-OST0001 complete [ 3301.211884] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 3312.352974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3318.107430] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3326.790867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.515067] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3335.511802] Lustre: 93458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3343.577553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3350.345916] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3351.150312] LustreError: 93813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3351.173914] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:161) [ 3351.175256] LustreError: 93813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 3357.685858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3554 to 0x280000401:3585) [ 3359.182068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3364.858853] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3361) [ 3364.886384] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:161) [ 3366.345832] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3375.357243] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3379.223655] Lustre: 95331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3386.673357] Lustre: DEBUG MARKER: == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 13:01:00 (1784048460) [ 3388.778397] Lustre: 95097:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 278, rollback = 2 [ 3388.785788] Lustre: 95097:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1361 previous similar messages [ 3388.789906] Lustre: 95097:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 3388.795751] Lustre: 95097:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1361 previous similar messages [ 3388.801858] Lustre: 95097:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3388.806819] Lustre: 95097:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1361 previous similar messages [ 3388.814366] Lustre: 95097:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3388.834971] Lustre: 95097:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1361 previous similar messages [ 3388.841985] Lustre: 95097:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 3388.847386] Lustre: 95097:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1361 previous similar messages [ 3388.851641] Lustre: 95097:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3388.857504] Lustre: 95097:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1361 previous similar messages [ 3393.105439] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3393.111659] Lustre: Skipped 63 previous similar messages [ 3419.617180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3419.631434] LustreError: Skipped 3 previous similar messages [ 3419.635594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3424.553978] Lustre: server umount lustre-MDT0000 complete [ 3428.879949] LustreError: 92297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048504 with bad export cookie 15014200346484653787 [ 3428.905972] LustreError: 92297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3429.232603] Lustre: server umount lustre-MDT0001 complete [ 3444.157711] Lustre: server umount lustre-OST0000 complete [ 3447.264231] Lustre: 16404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048507/real 1784048507] req@ffff8d35b7e17100 x1870707294430336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784048523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3448.624494] Lustre: server umount lustre-OST0001 complete [ 3467.089328] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 3478.515606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3479.047892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3479.050297] Lustre: Skipped 13 previous similar messages [ 3483.678457] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3492.993699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.776438] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3501.049900] Lustre: 99381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3507.692885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3514.306733] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3518.265851] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:193) [ 3522.622668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3523.842633] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:193) [ 3523.844933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3682 to 0x280000401:3713) [ 3523.854205] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3393) [ 3528.462849] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3535.664347] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3545.901551] Lustre: DEBUG MARKER: == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 13:03:40 (1784048620) [ 3549.390535] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3549.393380] Lustre: Skipped 63 previous similar messages [ 3549.564724] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3560.725459] Lustre: DEBUG MARKER: == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 13:03:54 (1784048634) [ 3562.873520] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3562.921803] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3562.925783] Lustre: Skipped 2 previous similar messages [ 3573.105626] Lustre: DEBUG MARKER: == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 13:04:07 (1784048647) [ 3574.821419] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_15c MDS newer than 2.7.55, LU-6475 [ 3576.658199] Lustre: DEBUG MARKER: == sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 13:04:11 (1784048651) [ 3582.577840] Lustre: *** cfs_fail_loc=1709, val=0*** [ 3582.580587] LustreError: 98247:0:(mdt_reint.c:2645:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240002340:0x6c:0x0]/f96 failed: rc = -5 [ 3646.950474] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.950895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3646.962517] Lustre: Skipped 9 previous similar messages [ 3646.967625] Lustre: Skipped 6 previous similar messages [ 3650.313854] Lustre: server umount lustre-MDT0000 complete [ 3658.772395] LustreError: 98220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048734 with bad export cookie 15014200346484668522 [ 3658.775292] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3658.795255] LustreError: 98220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3658.797118] LustreError: Skipped 1 previous similar message [ 3659.195310] Lustre: server umount lustre-MDT0001 complete [ 3678.390906] Lustre: server umount lustre-OST0000 complete [ 3678.693883] Lustre: 16404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048738/real 1784048738] req@ffff8d348657b480 x1870707295058304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784048754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3683.873626] Lustre: 16404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048743/real 1784048743] req@ffff8d348644dc00 x1870707295058816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784048759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3683.900928] Lustre: 16404:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3686.643106] Lustre: server umount lustre-OST0001 complete [ 3704.718791] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 3707.484232] Key type lgssc unregistered [ 3707.743384] LNet: 105084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3707.748585] LNetError: 105084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3707.769731] LNet: Removed LNI 192.168.202.155@tcp [ 3708.875384] Key type .llcrypt unregistered [ 3708.880351] Key type ._llcrypt unregistered [ 3736.065476] Key type ._llcrypt registered [ 3736.067832] Key type .llcrypt registered [ 3736.262735] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 3747.927437] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 3748.939481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3748.973682] alg: No test for adler32 (adler32-zlib) [ 3750.014477] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 3750.350502] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 3752.039183] Key type lgssc registered [ 3752.794975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3802.539769] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 3816.310716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3816.394431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.678787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3817.702798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3817.834717] Lustre: lustre-MDT0000: new disk, initializing [ 3817.957872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3817.983266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3822.539433] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3837.100322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.191741] Lustre: 109521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3837.217602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3837.220542] Lustre: Skipped 1 previous similar message [ 3837.276204] Lustre: lustre-MDT0001: new disk, initializing [ 3837.331817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3837.369169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3837.381794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3842.320217] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3846.897833] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3857.009846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3857.265751] Lustre: lustre-OST0000: new disk, initializing [ 3857.268971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3857.273750] Lustre: 111458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3857.374701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3861.296399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3861.301871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3861.387369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3863.813632] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3878.217737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3878.375905] Lustre: lustre-OST0001: new disk, initializing [ 3878.379965] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3878.384323] Lustre: 112482:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3878.444687] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3885.123994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3885.139992] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3885.257547] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3887.091670] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3899.082604] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3910.185633] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3916.541507] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 13:09:50 (1784048990) === [ 3924.488621] Lustre: DEBUG MARKER: == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 13:09:58 (1784048998) [ 3924.855657] Lustre: 109527:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 3924.865783] Lustre: 109527:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3924.873584] Lustre: 109527:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3924.879890] Lustre: 109527:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 3924.887437] Lustre: 109527:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 3924.901675] Lustre: 109527:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3925.402539] Lustre: 109527:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 3925.410823] Lustre: 109527:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 6 previous similar messages [ 3925.415459] Lustre: 109527:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 3925.422366] Lustre: 109527:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 3925.430116] Lustre: 109527:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3925.447484] Lustre: 109527:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 3925.454761] Lustre: 109527:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3925.462175] Lustre: 109527:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 3925.468678] Lustre: 109527:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 3925.478386] Lustre: 109527:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 3925.485497] Lustre: 109527:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3925.495562] Lustre: 109527:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 3926.410154] Lustre: 109527:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 3926.416254] Lustre: 109527:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 146 previous similar messages [ 3926.420991] Lustre: 109527:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 3926.426377] Lustre: 109527:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 146 previous similar messages [ 3926.432490] Lustre: 109527:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3926.441405] Lustre: 109527:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 146 previous similar messages [ 3926.469691] Lustre: 112672:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 7/369/0 [ 3926.475306] Lustre: 112672:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 149 previous similar messages [ 3926.482728] Lustre: 112672:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 3926.494266] Lustre: 112672:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 149 previous similar messages [ 3926.523449] Lustre: 112672:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3926.531366] Lustre: 112672:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 149 previous similar messages [ 3928.544298] Lustre: *** cfs_fail_loc=1613, val=0*** [ 3942.293861] Lustre: DEBUG MARKER: == sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 13:10:15 (1784049015) [ 3943.745444] Lustre: 112672:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 278, rollback = 2 [ 3943.751975] Lustre: 112672:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 155 previous similar messages [ 3943.765967] Lustre: 112672:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 3943.769943] Lustre: 112672:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 155 previous similar messages [ 3943.794738] Lustre: 112672:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3943.802269] Lustre: 112672:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 155 previous similar messages [ 3943.891609] Lustre: 112672:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3943.917847] Lustre: 112672:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 152 previous similar messages [ 3943.944054] Lustre: 112672:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 3943.956809] Lustre: 112672:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 152 previous similar messages [ 3943.983963] Lustre: 112672:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3943.992279] Lustre: 112672:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 152 previous similar messages [ 3945.610974] Lustre: *** cfs_fail_loc=1614, val=0*** [ 3950.540286] Lustre: 111449:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 275, rollback = 2 [ 3950.549488] Lustre: 111449:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 11 previous similar messages [ 3950.555576] Lustre: 111449:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3950.572349] Lustre: 111449:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3950.580370] Lustre: 111449:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 3/275/0 [ 3950.588981] Lustre: 111449:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3950.600161] Lustre: 111449:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 1/4/0, quota 7/225/0 [ 3950.620370] Lustre: 111449:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3950.631237] Lustre: 111449:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 3950.648050] Lustre: 111449:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3950.674020] Lustre: 111449:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3950.684893] Lustre: 111449:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3959.022755] Lustre: DEBUG MARKER: == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 13:10:33 (1784049033) [ 3959.519526] Lustre: 109527:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 3959.527033] Lustre: 109527:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 3959.532504] Lustre: 109527:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3959.547807] Lustre: 109527:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3959.560766] Lustre: 109527:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3959.577724] Lustre: 109527:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3959.587087] Lustre: 109527:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 3959.593647] Lustre: 109527:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3959.611176] Lustre: 109527:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 3959.620444] Lustre: 109527:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3959.639227] Lustre: 109527:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3959.649981] Lustre: 109527:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3962.493920] Lustre: *** cfs_fail_loc=1615, val=0*** [ 3962.499270] Lustre: Skipped 1 previous similar message [ 3980.997919] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_18b skipping excluded test 18b [ 3982.772235] Lustre: DEBUG MARKER: == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 13:10:57 (1784049057) [ 3983.125981] Lustre: 112672:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 3983.132340] Lustre: 112672:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 21 previous similar messages [ 3983.136819] Lustre: 112672:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 3983.142070] Lustre: 112672:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3983.157296] Lustre: 112672:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3983.161839] Lustre: 112672:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3983.167737] Lustre: 112672:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3983.172762] Lustre: 112672:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3983.177861] Lustre: 112672:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 3983.181752] Lustre: 112672:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3983.186914] Lustre: 112672:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3983.192133] Lustre: 112672:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3985.710618] Lustre: *** cfs_fail_loc=1617, val=0*** [ 3985.777877] Lustre: *** cfs_fail_loc=1617, val=0*** [ 3988.357045] Lustre: *** cfs_fail_loc=1616, val=0*** [ 3988.358819] Lustre: Skipped 5 previous similar messages [ 4007.073483] Lustre: DEBUG MARKER: == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 13:11:21 (1784049081) [ 4009.636988] Lustre: *** cfs_fail_loc=1618, val=0*** [ 4009.642992] Lustre: Skipped 5 previous similar messages [ 4044.261157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4044.265460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4044.276410] Lustre: Skipped 3 previous similar messages [ 4044.286486] Lustre: Skipped 3 previous similar messages [ 4049.379018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4049.389940] Lustre: Skipped 1 previous similar message [ 4049.855810] Lustre: server umount lustre-MDT0000 complete [ 4054.072222] LustreError: 109513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049129 with bad export cookie 3467391763849342499 [ 4054.075765] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4054.079337] LustreError: 109513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4054.473476] Lustre: server umount lustre-MDT0001 complete [ 4068.727422] Lustre: server umount lustre-OST0000 complete [ 4070.880170] Lustre: 106679:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784049130/real 1784049130] req@ffff8d35b7343480 x1870710781982976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784049146 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4070.897342] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.673612] Lustre: server umount lustre-OST0001 complete [ 4088.558775] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 4099.954575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.319398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.064576] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4105.696566] LustreError: 118200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4105.717958] LustreError: 118200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4110.817086] LustreError: 118201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.978661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4111.142785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4114.518363] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4116.752994] Lustre: 119340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4123.734493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4125.171594] LustreError: 119694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.216199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:116 to 0x280000401:161) [ 4130.282780] LustreError: 119694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4133.040904] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4135.393026] LustreError: 119745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4135.429899] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:33) [ 4143.944683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4144.139726] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4144.143640] Lustre: Skipped 1 previous similar message [ 4149.233633] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:33) [ 4149.241849] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 4151.631796] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4160.510206] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4164.378925] Lustre: 121212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4179.080324] Lustre: DEBUG MARKER: == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 13:14:13 (1784049253) [ 4179.423682] Lustre: 118197:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 4179.432923] Lustre: 118197:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 46 previous similar messages [ 4179.441097] Lustre: 118197:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4179.446112] Lustre: 118197:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4179.450728] Lustre: 118197:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4179.455701] Lustre: 118197:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4179.463313] Lustre: 118197:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 4179.469146] Lustre: 118197:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4179.479766] Lustre: 118197:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 4179.486186] Lustre: 118197:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4179.492082] Lustre: 118197:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4179.498331] Lustre: 118197:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4181.382618] Lustre: *** cfs_fail_loc=1618, val=0*** [ 4181.386633] Lustre: Skipped 3 previous similar messages [ 4215.776352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.780120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4215.786126] Lustre: Skipped 2 previous similar messages [ 4215.788922] Lustre: Skipped 1 previous similar message [ 4220.224105] Lustre: server umount lustre-MDT0000 complete [ 4220.896437] LustreError: 118201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.910643] LustreError: 118201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4224.139095] LustreError: 118942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049300 with bad export cookie 3467391763849357766 [ 4224.142969] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4224.154519] LustreError: 118942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4224.413257] Lustre: server umount lustre-MDT0001 complete [ 4238.747815] Lustre: server umount lustre-OST0000 complete [ 4241.183099] Lustre: 106682:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784049301/real 1784049301] req@ffff8d35baae2300 x1870710782062848/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784049317 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4241.208097] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.226824] Lustre: Skipped 1 previous similar message [ 4242.479086] Lustre: server umount lustre-OST0001 complete [ 4258.090777] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 4267.932234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4268.380680] LustreError: 123781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4268.506455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.672129] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4282.364363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4288.147283] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4291.554609] Lustre: 124921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4298.471020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4304.749449] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4307.946491] LustreError: 125275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.955299] LustreError: 125275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4307.959658] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:65) [ 4309.806794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4313.425552] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4314.018579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4314.019514] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:65) [ 4314.020060] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 4317.652219] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4319.730453] Lustre: 126788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4325.112681] Lustre: 124513:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 252 < left 260, rollback = 2 [ 4325.115557] Lustre: 124513:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 21 previous similar messages [ 4325.118196] Lustre: 124513:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4325.120102] Lustre: 124513:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4325.122183] Lustre: 124513:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/260/0 [ 4325.124634] Lustre: 124513:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4325.126474] Lustre: 124513:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 0/0/0, quota 4/150/2 [ 4325.128379] Lustre: 124513:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4325.130846] Lustre: 124513:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 1/17/2, delete: 0/0/0 [ 4325.133027] Lustre: 124513:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4325.135294] Lustre: 124513:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4325.141806] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4325.142364] Lustre: 124513:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4353.628240] Lustre: DEBUG MARKER: == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 13:17:07 (1784049427) [ 4357.490411] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4357.493066] Lustre: Skipped 3 previous similar messages [ 4364.593453] Lustre: *** cfs_fail_loc=161c, val=0*** [ 4382.591141] Lustre: DEBUG MARKER: == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) ========================================================== 13:17:37 (1784049457) [ 4385.307329] Lustre: *** cfs_fail_loc=162e, val=0*** [ 4385.310258] Lustre: *** cfs_fail_loc=162e, val=0*** [ 4385.313205] Lustre: Skipped 7 previous similar messages [ 4400.143172] Lustre: DEBUG MARKER: == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 13:17:54 (1784049474) [ 4421.155701] Lustre: DEBUG MARKER: == sanity-lfsck test 19a: OST-object inconsistency self detect ========================================================== 13:18:15 (1784049495) [ 4434.082444] Lustre: DEBUG MARKER: == sanity-lfsck test 19b: OST-object inconsistency self repair ========================================================== 13:18:28 (1784049508) [ 4437.132100] Lustre: *** cfs_fail_loc=1611, val=0*** [ 4437.161576] Lustre: *** cfs_fail_loc=1611, val=0*** [ 4437.163502] Lustre: Skipped 3 previous similar messages [ 4442.803270] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.55@tcp inode [0x2000013a1:0x1f:0x0] object 0x280000401:203 extent [0-4095], client returned csum 0 (type 4), server csum 703fb3ad (type 4) [ 4443.974350] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.55@tcp inode [0x2000013a1:0x20:0x0] object 0x280000401:204 extent [0-4095], client returned csum 0 (type 4), server csum 7cbb935c (type 4) [ 4452.965861] Lustre: DEBUG MARKER: == sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================================================== 13:18:47 (1784049527) [ 4453.279792] Lustre: 127235:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 4453.291252] Lustre: 127235:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 104 previous similar messages [ 4453.296762] Lustre: 127235:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4453.306530] Lustre: 127235:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 4453.315231] Lustre: 127235:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4453.322741] Lustre: 127235:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 4453.328767] Lustre: 127235:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 4453.333231] Lustre: 127235:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 4453.339146] Lustre: 127235:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 4453.344698] Lustre: 127235:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 4453.351521] Lustre: 127235:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4453.358774] Lustre: 127235:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 4455.290630] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4455.292174] Lustre: Skipped 3 previous similar messages [ 4481.133531] Lustre: DEBUG MARKER: == sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case ========================================================== 13:19:15 (1784049555) [ 4488.911970] Lustre: DEBUG MARKER: == sanity-lfsck test 21: run all LFSCK components by default ========================================================== 13:19:22 (1784049562) [ 4504.063366] Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) ========================================================== 13:19:38 (1784049578) [ 4506.505448] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4506.512498] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4506.517452] Lustre: Skipped 1 previous similar message [ 4520.802413] Lustre: DEBUG MARKER: == sanity-lfsck test 22b: LFSCK can repair unmatched pairs (2) ========================================================== 13:19:54 (1784049594) [ 4522.899432] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4522.903525] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4534.597434] Lustre: DEBUG MARKER: == sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) ========================================================== 13:20:08 (1784049608) [ 4536.506193] Lustre: *** cfs_fail_loc=1620, val=0*** [ 4552.202830] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_23b skipping excluded test 23b [ 4553.873310] Lustre: DEBUG MARKER: == sanity-lfsck test 23c: LFSCK can repair dangling name entry (3) ========================================================== 13:20:28 (1784049628) [ 4559.828266] Lustre: *** cfs_fail_loc=1621, val=127*** [ 4559.835166] Lustre: Skipped 1 previous similar message [ 4562.834393] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4586.202924] Lustre: DEBUG MARKER: == sanity-lfsck test 23d: LFSCK can repair a dangling name entry to a remote object ========================================================== 13:21:00 (1784049660) [ 4588.871270] Lustre: Failing over lustre-MDT0000 [ 4589.436689] Lustre: server umount lustre-MDT0000 complete [ 4590.051952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4590.060372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4590.080985] LustreError: 124513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.319339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4599.472065] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4599.770698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4599.782326] Lustre: Skipped 3 previous similar messages [ 4599.822624] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4602.107599] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4604.909678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4604.970514] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4605.064250] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:130 to 0x2c0000401:161) [ 4605.066797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 4605.307859] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4607.512518] LustreError: 123778:0:(mdt_open.c:1322:mdt_cross_open()) lustre-MDT0000: [0x2000013a3:0x82:0x0] doesn't exist!: rc = -14 [ 4619.066955] Lustre: DEBUG MARKER: == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ========================================================== 13:21:33 (1784049693) [ 4621.250557] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4621.377317] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4621.379982] Lustre: Skipped 1 previous similar message [ 4633.863868] Lustre: DEBUG MARKER: == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ========================================================== 13:21:47 (1784049707) [ 4636.252073] Lustre: *** cfs_fail_loc=1623, val=0*** [ 4649.460633] Lustre: DEBUG MARKER: == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace ========================================================== 13:22:03 (1784049723) [ 4651.221160] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4664.145518] Lustre: DEBUG MARKER: == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========================================================== 13:22:18 (1784049738) [ 4677.780572] Lustre: DEBUG MARKER: == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan ========================================================== 13:22:31 (1784049751) [ 4679.946827] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4679.960916] Lustre: Skipped 1 previous similar message [ 4697.520309] Lustre: DEBUG MARKER: == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ========================================================== 13:22:50 (1784049770) [ 4711.157412] Lustre: DEBUG MARKER: == sanity-lfsck test 28: Skip the failed MDT(s) when handle orphan MDT-objects ========================================================== 13:23:05 (1784049785) [ 4711.479460] Lustre: 123776:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 4711.485784] Lustre: 123776:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 497 previous similar messages [ 4711.491810] Lustre: 123776:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4711.496309] Lustre: 123776:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4711.499741] Lustre: 123776:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4711.504967] Lustre: 123776:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4711.510755] Lustre: 123776:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 4711.517347] Lustre: 123776:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4711.522955] Lustre: 123776:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 4711.528083] Lustre: 123776:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4711.536080] Lustre: 123776:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4711.541878] Lustre: 123776:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4714.450361] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4714.456684] Lustre: Skipped 2 previous similar messages [ 4718.824256] Lustre: *** cfs_fail_loc=161c, val=0*** [ 4718.828235] Lustre: Skipped 1 previous similar message [ 4739.347912] Lustre: DEBUG MARKER: == sanity-lfsck test 29b: LFSCK can repair bad nlink count (2) ========================================================== 13:23:33 (1784049813) [ 4756.335553] Lustre: DEBUG MARKER: == sanity-lfsck test 29c: verify linkEA size limitation == 13:23:49 (1784049829) [ 4786.371819] Lustre: DEBUG MARKER: == sanity-lfsck test 29d: accessing non-existing inode shouldn't turn fs read-only (ldiskfs) ========================================================== 13:24:20 (1784049860) [ 4788.309825] Lustre: *** cfs_fail_loc=1626, val=0*** [ 4788.316083] Lustre: Skipped 2 previous similar messages [ 4790.405419] LustreError: 125294:0:(osd_handler.c:284:osd_idc_find_or_init()) lustre-MDT0000: cannot lookup FID [0x2000013a3:0x9e:0x0]: rc = -2 [ 4802.027538] Lustre: DEBUG MARKER: == sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 13:24:35 (1784049875) [ 4817.882927] Lustre: Failing over lustre-MDT0000 [ 4818.607456] Lustre: server umount lustre-MDT0000 complete [ 4819.938910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4819.950807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4819.967987] LustreError: 123776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4819.982074] Lustre: Skipped 5 previous similar messages [ 4820.027416] LustreError: 123776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 4834.801367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4834.982688] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4835.264857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4835.310187] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4840.422675] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4840.431363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4840.456496] Lustre: Skipped 3 previous similar messages [ 4840.492674] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4840.500373] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4840.557694] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:167 to 0x2c0000401:193) [ 4840.559595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:294 to 0x280000401:321) [ 4856.472284] Lustre: DEBUG MARKER: == sanity-lfsck test 31a: The LFSCK can find/repair the name entry with bad name hash (1) ========================================================== 13:25:30 (1784049930) [ 4872.424403] Lustre: DEBUG MARKER: == sanity-lfsck test 31b: The LFSCK can find/repair the name entry with bad name hash (2) ========================================================== 13:25:46 (1784049946) [ 4889.881695] Lustre: DEBUG MARKER: == sanity-lfsck test 31c: Re-generate the lost master LMV EA for striped directory ========================================================== 13:26:03 (1784049963) [ 4891.576626] Lustre: *** cfs_fail_loc=1629, val=0*** [ 4891.580981] Lustre: Skipped 5 previous similar messages [ 4920.946497] Lustre: DEBUG MARKER: == sanity-lfsck test 31d: Set broken striped directory (modified after broken) as read-only ========================================================== 13:26:35 (1784049995) [ 4926.640179] Lustre: Failing over lustre-MDT0000 [ 4926.916643] Lustre: server umount lustre-MDT0000 complete [ 4927.457658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4927.460406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.473051] Lustre: Skipped 1 previous similar message [ 4935.235261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4935.417829] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4935.705079] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4939.700228] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4940.778971] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4940.786966] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4940.799116] Lustre: Skipped 3 previous similar messages [ 4940.832633] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4940.867735] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:167 to 0x2c0000401:225) [ 4940.871604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:294 to 0x280000401:353) [ 4950.658452] Lustre: Failing over lustre-MDT0000 [ 4950.930546] Lustre: server umount lustre-MDT0000 complete [ 4960.040411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4960.171961] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4960.471596] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4963.063302] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4965.287393] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4965.859126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4965.870864] Lustre: Skipped 3 previous similar messages [ 4965.917776] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4965.986100] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:167 to 0x2c0000401:257) [ 4965.994696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4976.433953] Lustre: DEBUG MARKER: == sanity-lfsck test 31e: Re-generate the lost slave LMV EA for striped directory (1) ========================================================== 13:27:30 (1784050050) [ 4990.133257] Lustre: DEBUG MARKER: == sanity-lfsck test 31f: Re-generate the lost slave LMV EA for striped directory (2) ========================================================== 13:27:44 (1784050064) [ 5004.039330] Lustre: DEBUG MARKER: == sanity-lfsck test 31g: Repair the corrupted slave LMV EA ========================================================== 13:27:57 (1784050077) [ 5047.089785] Lustre: DEBUG MARKER: == sanity-lfsck test 31h: Repair the corrupted shard's name entry ========================================================== 13:28:40 (1784050120) [ 5049.382520] Lustre: *** cfs_fail_loc=162c, val=0*** [ 5049.384612] Lustre: Skipped 13 previous similar messages [ 5065.631774] Lustre: DEBUG MARKER: == sanity-lfsck test 32a: stop LFSCK when some OST failed ========================================================== 13:29:00 (1784050140) [ 5075.805389] LustreError: 148140:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5078.690191] Lustre: Failing over lustre-OST0000 [ 5078.837198] Lustre: server umount lustre-OST0000 complete [ 5078.872483] LustreError: 148140:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 5078.886066] LustreError: 148140:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5079.012038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5079.019637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5079.037492] Lustre: Skipped 7 previous similar messages [ 5079.048376] LustreError: 125276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5079.076957] LustreError: 125276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 5081.767063] LustreError: 148140:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout interrupted [ 5095.050048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5095.449542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5095.455073] Lustre: Skipped 2 previous similar messages [ 5095.468901] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5096.936876] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5096.979557] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5096.979893] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5097.006366] Lustre: Skipped 3 previous similar messages [ 5103.607781] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5114.141546] Lustre: DEBUG MARKER: == sanity-lfsck test 32b: stop LFSCK when some MDT failed ========================================================== 13:29:48 (1784050188) [ 5128.534841] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 5144.708045] Lustre: 150945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5169.552389] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5174.041214] Lustre: 152079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5188.129106] LustreError: 152219:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5191.160847] LustreError: 152219:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 5191.166305] Lustre: Failing over lustre-MDT0001 [ 5191.185028] LustreError: 152219:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5191.202123] LustreError: 152219:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) Skipped 1 previous similar message [ 5191.258199] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 5191.267525] LustreError: Skipped 1 previous similar message [ 5191.271544] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.279469] Lustre: Skipped 1 previous similar message [ 5191.285773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5191.290138] Lustre: Skipped 2 previous similar messages [ 5192.687788] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5194.239122] LustreError: 152219:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 5194.253855] LustreError: 152219:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) Skipped 1 previous similar message [ 5194.639306] Lustre: server umount lustre-MDT0001 complete [ 5210.127513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5210.618213] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5215.736947] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5215.749478] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5215.760160] Lustre: Skipped 1 previous similar message [ 5215.787594] Lustre: lustre-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5215.861569] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:71 to 0x280000400:97) [ 5215.862949] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:68 to 0x2c0000400:97) [ 5216.317737] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5225.253946] Lustre: DEBUG MARKER: == sanity-lfsck test 33: check LFSCK paramters =========== 13:31:39 (1784050299) [ 5239.822286] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 5259.936583] Lustre: 154939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5283.107745] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5287.264237] Lustre: 156075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5289.649639] Lustre: 123777:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 5289.663757] Lustre: 123777:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1241 previous similar messages [ 5289.674419] Lustre: 123777:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 5289.682582] Lustre: 123777:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1241 previous similar messages [ 5289.689975] Lustre: 123777:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5289.698224] Lustre: 123777:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1241 previous similar messages [ 5289.707448] Lustre: 123777:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 5289.715698] Lustre: 123777:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1241 previous similar messages [ 5289.727138] Lustre: 123777:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 5289.744709] Lustre: 123777:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1241 previous similar messages [ 5289.758937] Lustre: 123777:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5289.771366] Lustre: 123777:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1241 previous similar messages [ 5314.538351] Lustre: DEBUG MARKER: == sanity-lfsck test 34: LFSCK can rebuild the lost agent object ========================================================== 13:33:08 (1784050388) [ 5316.240892] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_34 Only valid for ZFS backend [ 5318.090856] Lustre: DEBUG MARKER: == sanity-lfsck test 35: LFSCK can rebuild the lost agent entry ========================================================== 13:33:12 (1784050392) [ 5325.898302] Lustre: *** cfs_fail_loc=1631, val=0*** [ 5338.592129] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5338.603016] Lustre: Skipped 2 previous similar messages [ 5338.604218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5339.330932] Lustre: server umount lustre-MDT0000 complete [ 5344.560432] LustreError: 123763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784050420 with bad export cookie 3467391763849430601 [ 5344.563190] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5344.578863] LustreError: 123763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5345.466424] Lustre: server umount lustre-MDT0001 complete [ 5360.860393] Lustre: server umount lustre-OST0000 complete [ 5364.192938] Lustre: 106679:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784050424/real 1784050424] req@ffff8d3485073480 x1870710783469952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784050440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5365.224245] Lustre: 106681:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784050425/real 1784050425] req@ffff8d348652f480 x1870710783470336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784050441 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5365.975699] Lustre: server umount lustre-OST0001 complete [ 5382.355995] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 5393.091431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5397.924631] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5406.330808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.600771] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5414.829290] Lustre: 159970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5422.073985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5428.672441] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5431.470027] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:71 to 0x280000400:129) [ 5437.633130] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5439.928415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:540 to 0x280000401:577) [ 5439.948682] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:412 to 0x2c0000401:449) [ 5439.956890] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:68 to 0x2c0000400:129) [ 5444.761829] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5454.007784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5458.488767] Lustre: 161840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5469.254264] Lustre: DEBUG MARKER: == sanity-lfsck test 36a: rebuild LOV EA for mirrored file (1) ========================================================== 13:35:43 (1784050543) [ 5471.636884] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36a needs >= 3 OSTs [ 5473.683277] Lustre: DEBUG MARKER: == sanity-lfsck test 36b: rebuild LOV EA for mirrored file (2) ========================================================== 13:35:47 (1784050547) [ 5475.419317] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36b needs >= 3 OSTs [ 5477.812176] Lustre: DEBUG MARKER: == sanity-lfsck test 36c: rebuild LOV EA for mirrored file (3) ========================================================== 13:35:51 (1784050551) [ 5479.935679] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36c needs >= 3 OSTs [ 5482.493860] Lustre: DEBUG MARKER: == sanity-lfsck test 37: LFSCK must skip a ORPHAN ======== 13:35:56 (1784050556) [ 5495.910199] Lustre: DEBUG MARKER: == sanity-lfsck test 38: LFSCK does not break foreign file and reverse is also true ========================================================== 13:36:10 (1784050570) [ 5509.146690] Lustre: DEBUG MARKER: == sanity-lfsck test 39: LFSCK does not break foreign dir and reverse is also true ========================================================== 13:36:23 (1784050583) [ 5526.790478] Lustre: DEBUG MARKER: == sanity-lfsck test 40a: LFSCK correctly fixes lmm_oi in composite layout ========================================================== 13:36:40 (1784050600) [ 5542.423714] Lustre: DEBUG MARKER: == sanity-lfsck test 41: SEL support in LFSCK ============ 13:36:56 (1784050616) [ 5568.252763] Lustre: DEBUG MARKER: == sanity-lfsck test 42: LFSCK repairs inconsistent MDT-object/OST-object encryption flags ========================================================== 13:37:22 (1784050642) [ 5604.155528] Lustre: *** cfs_fail_loc=1632, val=0*** [ 5620.061717] Lustre: DEBUG MARKER: == sanity-lfsck test 43: LFSCK does not loop endlessly on iget failure in scanning-phase1 ========================================================== 13:38:14 (1784050694) [ 5623.002678] Lustre: Failing over lustre-MDT0001 [ 5623.193357] Lustre: server umount lustre-MDT0001 complete [ 5624.289413] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.291291] LustreError: 162109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5624.298207] Lustre: Skipped 3 previous similar messages [ 5624.319118] LustreError: 162109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 31 previous similar messages [ 5631.515894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.880617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5631.884121] Lustre: Skipped 5 previous similar messages [ 5631.917164] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5631.917988] Lustre: lustre-MDT0001: Aborting client recovery [ 5631.928754] LustreError: 165637:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0001: Aborting recovery [ 5631.936783] Lustre: 165661:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5631.943883] Lustre: 165661:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 528d79c2-2e3d-4369-ba00-5dc47bc82808@ [ 5631.951948] Lustre: lustre-MDT0001: disconnecting 2 stale clients [ 5631.961171] Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] [ 5631.972941] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] [ 5632.018433] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:71 to 0x280000400:161) [ 5632.022629] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:131 to 0x2c0000400:161) [ 5637.111167] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5637.122382] Lustre: Skipped 2 previous similar messages [ 5637.142581] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5637.568342] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5641.701605] Lustre: *** cfs_fail_loc=1e2, val=483*** [ 5646.774932] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid [ 5647.240480] Lustre: DEBUG MARKER: osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid in FULL state after 0 sec [ 5653.073849] Lustre: DEBUG MARKER: == sanity-lfsck test 44: umount while lfsck is stopping == 13:38:47 (1784050727) [ 5661.863293] Lustre: *** cfs_fail_loc=1600, val=3*** [ 5664.723717] Lustre: Failing over lustre-MDT0000 [ 5665.154387] Lustre: server umount lustre-MDT0000 complete [ 5667.825665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5675.341676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.538974] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5675.932203] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5675.952220] Lustre: Skipped 2 previous similar messages [ 5678.324874] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5681.126920] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5681.142672] Lustre: Skipped 2 previous similar messages [ 5681.203914] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5681.341086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:621 to 0x280000401:641) [ 5681.341467] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 5683.029499] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5697.365995] Lustre: DEBUG MARKER: == sanity-lfsck test 45: LFSCK should fix UID/GID/PROJID of OST object ========================================================== 13:39:30 (1784050770) [ 5731.097143] Lustre: Failing over lustre-OST0000 [ 5731.290449] Lustre: server umount lustre-OST0000 complete [ 5739.614240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5753.762123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5753.999652] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5755.124580] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5755.187522] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5755.188282] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5755.213607] Lustre: Skipped 3 previous similar messages [ 5762.257025] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5771.270246] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5771.668372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5779.816464] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5780.241626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5788.219536] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff9f7188394000.ost_server_uuid 50 [ 5846.083263] Lustre: DEBUG MARKER: rpc test_45: @@@@@@ FAIL: can't put import for osc.lustre-OST0000-osc-ffff9f7188394000.ost_server_uuid into FULL state after 50 sec, have IDLE