[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488862294 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002344] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006675] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.010103] LSM: Security Framework initializing [ 0.011063] Yama: becoming mindful. [ 0.012044] SELinux: Initializing. [ 0.013084] *** VALIDATE selinux *** [ 0.022305] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027516] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029100] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031106] *** VALIDATE tmpfs *** [ 0.033144] *** VALIDATE proc *** [ 0.034222] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036266] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043343] debug: unmapping init [mem 0xffffffffa9e59000-0xffffffffa9e60fff] [ 0.045163] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046693] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050009] ... value mask: 0000ffffffffffff [ 0.051012] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054308] rcu: Hierarchical SRCU implementation. [ 0.056469] smp: Bringing up secondary CPUs ... [ 0.057587] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.062149] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139016] node 0 deferred pages initialised in 70ms [ 0.143008] devtmpfs: initialized [ 0.144000] x86/mm: Memory block size: 128MB [ 0.147199] gcov: version magic: 0x41383552 [ 0.150229] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152251] pinctrl core: initialized pinctrl subsystem [ 0.153166] [ 0.153723] ************************************************************* [ 0.154013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155013] ** ** [ 0.156013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157017] ** ** [ 0.158012] ** This means that this kernel is built to expose internal ** [ 0.159012] ** IOMMU data structures, which may compromise security on ** [ 0.160014] ** your system. ** [ 0.161012] ** ** [ 0.162015] ** If you see this message and you are not debugging the ** [ 0.163013] ** kernel, report this immediately to your vendor! ** [ 0.164010] ** ** [ 0.165013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166014] ************************************************************* [ 0.167647] NET: Registered protocol family 16 [ 0.168430] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173136] cpuidle: using governor menu [ 0.175704] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179542] PCI: Using configuration type 1 for base access [ 0.183110] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193147] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196053] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200179] cryptd: max_cpu_qlen set to 1000 [ 0.204281] ACPI: Added _OSI(Module Device) [ 0.206018] ACPI: Added _OSI(Processor Device) [ 0.208014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215358] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.223466] ACPI: Interpreter enabled [ 0.224055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.225010] ACPI: Using IOAPIC for interrupt routing [ 0.226093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227451] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237898] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.239042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244061] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250353] acpiphp: Slot [2] registered [ 0.252165] acpiphp: Slot [5] registered [ 0.253115] acpiphp: Slot [6] registered [ 0.254111] acpiphp: Slot [7] registered [ 0.256092] acpiphp: Slot [8] registered [ 0.257118] acpiphp: Slot [9] registered [ 0.259133] acpiphp: Slot [10] registered [ 0.262072] acpiphp: Slot [3] registered [ 0.264105] acpiphp: Slot [4] registered [ 0.265159] acpiphp: Slot [11] registered [ 0.267121] acpiphp: Slot [12] registered [ 0.268095] acpiphp: Slot [13] registered [ 0.270161] acpiphp: Slot [14] registered [ 0.272091] acpiphp: Slot [15] registered [ 0.273091] acpiphp: Slot [16] registered [ 0.274067] acpiphp: Slot [17] registered [ 0.275075] acpiphp: Slot [18] registered [ 0.277104] acpiphp: Slot [19] registered [ 0.279170] acpiphp: Slot [20] registered [ 0.280114] acpiphp: Slot [21] registered [ 0.282078] acpiphp: Slot [22] registered [ 0.284096] acpiphp: Slot [23] registered [ 0.286122] acpiphp: Slot [24] registered [ 0.287095] acpiphp: Slot [25] registered [ 0.289101] acpiphp: Slot [26] registered [ 0.291106] acpiphp: Slot [27] registered [ 0.292155] acpiphp: Slot [28] registered [ 0.294113] acpiphp: Slot [29] registered [ 0.295106] acpiphp: Slot [30] registered [ 0.297109] acpiphp: Slot [31] registered [ 0.298098] PCI host bridge to bus 0000:00 [ 0.299018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.301018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.303022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.306022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.309023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.312027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.314224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.317112] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.319302] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.330014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.335089] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.337014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.339013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.341014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.343443] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.346731] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.348042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.351852] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.357016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.371014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.377025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.383103] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.388013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.395027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.416015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.427470] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.434014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.445012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.469030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.481543] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.494018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.505021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.523015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.534868] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.546024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.559022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.579024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.590000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.605019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.611016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.634029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.643950] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.653024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.662021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.680026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.690786] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.694370] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.696385] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.699432] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.702446] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.707269] iommu: Default domain type: Passthrough [ 0.708329] SCSI subsystem initialized [ 0.709126] ACPI: bus type USB registered [ 0.711177] usbcore: registered new interface driver usbfs [ 0.713126] usbcore: registered new interface driver hub [ 0.715173] usbcore: registered new device driver usb [ 0.717180] pps_core: LinuxPPS API ver. 1 registered [ 0.719019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.723124] PTP clock support registered [ 0.725183] EDAC MC: Ver: 3.0.0 [ 0.728360] PCI: Using ACPI for IRQ routing [ 0.731146] NetLabel: Initializing [ 0.732015] NetLabel: domain hash size = 128 [ 0.734017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.736112] NetLabel: unlabeled traffic allowed by default [ 0.739205] vgaarb: loaded [ 0.741307] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.743017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.749418] clocksource: Switched to clocksource kvm-clock [ 0.861904] VFS: Disk quotas dquot_6.6.0 [ 0.864048] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.867176] *** VALIDATE ramfs *** [ 0.868757] *** VALIDATE hugetlbfs *** [ 0.870914] pnp: PnP ACPI init [ 0.873491] pnp: PnP ACPI: found 6 devices [ 0.894744] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.898519] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.900858] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.903575] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.906570] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.909467] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.912956] NET: Registered protocol family 2 [ 0.915610] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.921091] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.925402] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.929775] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.933987] TCP: Hash tables configured (established 65536 bind 65536) [ 0.937373] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.940794] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.944060] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.947421] NET: Registered protocol family 1 [ 0.951146] RPC: Registered named UNIX socket transport module. [ 0.953570] RPC: Registered udp transport module. [ 0.955535] RPC: Registered tcp transport module. [ 0.957388] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.960023] NET: Registered protocol family 44 [ 0.962067] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.964347] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.966281] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.969052] PCI: CLS 0 bytes, default 64 [ 0.971089] Unpacking initramfs... [ 2.423522] debug: unmapping init [mem 0xffff9fd37cc54000-0xffff9fd37ffbffff] [ 2.427420] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.431216] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.435102] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.998642] Initialise system trusted keyrings [ 3.002202] Key type blacklist registered [ 3.004295] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.013664] zbud: loaded [ 3.016580] *** VALIDATE nfs *** [ 3.017884] *** VALIDATE nfs4 *** [ 3.019619] pstore: using deflate compression [ 3.023141] Platform Keyring initialized [ 3.122926] NET: Registered protocol family 38 [ 3.124814] Key type asymmetric registered [ 3.126537] Asymmetric key parser 'x509' registered [ 3.128476] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.132250] io scheduler mq-deadline registered [ 3.134193] io scheduler kyber registered [ 3.135898] io scheduler bfq registered [ 3.137926] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.140926] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.143790] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.146696] ACPI: Power Button [PWRF] [ 3.153645] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.161311] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.176236] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.184843] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.207235] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.233904] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.263451] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.269278] Non-volatile memory driver v1.3 [ 3.271260] Linux agpgart interface v0.103 [ 3.304092] virtio_blk virtio1: [vda] 145928 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.308818] vda: detected capacity change from 0 to 74715136 [ 3.326724] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.329956] vdb: detected capacity change from 0 to 1073741824 [ 3.347145] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.350599] vdc: detected capacity change from 0 to 2621440000 [ 3.367528] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.370094] vdd: detected capacity change from 0 to 2621440000 [ 3.386259] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.389746] vde: detected capacity change from 0 to 4294967296 [ 3.404766] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.407973] vdf: detected capacity change from 0 to 4294967296 [ 3.418246] libphy: Fixed MDIO Bus: probed [ 3.423331] usbcore: registered new interface driver usbserial_generic [ 3.426453] usbserial: USB Serial support registered for generic [ 3.429166] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.434502] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.436809] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.439406] mousedev: PS/2 mouse device common for all mice [ 3.442935] rtc_cmos 00:05: RTC can wake from S4 [ 3.446450] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.447555] rtc_cmos 00:05: registered as rtc0 [ 3.453631] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.454633] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.456027] intel_pstate: CPU model not supported [ 3.459484] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.463206] hid: raw HID events driver (C) Jiri Kosina [ 3.465324] usbcore: registered new interface driver usbhid [ 3.467765] usbhid: USB HID core driver [ 3.469471] drop_monitor: Initializing network drop monitor service [ 3.473018] Initializing XFRM netlink socket [ 3.475381] NET: Registered protocol family 10 [ 3.480526] Segment Routing with IPv6 [ 3.482329] NET: Registered protocol family 17 [ 3.484574] mpls_gso: MPLS GSO support [ 3.493661] RAS: Correctable Errors collector initialized. [ 3.496067] AVX version of gcm_enc/dec engaged. [ 3.497973] AES CTR mode by8 optimization enabled [ 3.578647] sched_clock: Marking stable (3578556720, 0)->(4549054923, -970498203) [ 3.582886] registered taskstats version 1 [ 3.585547] Loading compiled-in X.509 certificates [ 3.587718] zswap: loaded using pool lzo/zbud [ 3.612712] Key type big_key registered [ 3.624704] Key type encrypted registered [ 3.626388] ima: No TPM chip found, activating TPM-bypass! [ 3.628696] ima: Allocated hash algorithm: sha1 [ 3.630741] ima: No architecture policies found [ 3.632682] evm: Initialising EVM extended attributes: [ 3.634578] evm: security.selinux [ 3.635857] evm: security.ima [ 3.636975] evm: security.capability [ 3.638486] evm: HMAC attrs: 0x1 [ 3.641473] rtc_cmos 00:05: setting system clock to 2026-08-07 11:16:28 UTC (1786101388) [ 3.648252] debug: unmapping init [mem 0xffffffffaae03000-0xffffffffaaffffff] [ 3.651412] debug: unmapping init [mem 0xffffffffa9b82000-0xffffffffa9e58fff] [ 3.661077] Write protecting the kernel read-only data: 28672k [ 3.665203] debug: unmapping init [mem 0xffffffffa8203000-0xffffffffa83fffff] [ 3.668397] debug: unmapping init [mem 0xffffffffa8b14000-0xffffffffa8bfffff] [ 3.701462] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.711113] systemd[1]: Detected virtualization kvm. [ 3.712954] systemd[1]: Detected architecture x86-64. [ 3.714804] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.738315] systemd[1]: No hostname configured. [ 3.740113] systemd[1]: Set hostname to . [ 3.741664] random: systemd: uninitialized urandom read (16 bytes read) [ 3.743838] systemd[1]: Initializing machine ID from random generator. [ 3.792792] random: ln: uninitialized urandom read (6 bytes read) [ 3.895636] random: systemd: uninitialized urandom read (16 bytes read) [ 3.898940] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.903939] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.910638] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.508464] device-mapper: uevent: version 1.0.3 [ 4.513918] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.872311] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.181727] virtio_net virtio0 ens2: renamed from eth0 [ 5.257336] scsi host0: ata_piix [ 5.278253] scsi host1: ata_piix [ 5.281741] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.284717] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.443287] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.145639] random: crng init done [ 10.147163] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.481529] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.590796] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.838753] SELinux: Disabled at runtime. [ 11.899555] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.909177] systemd[1]: Detected virtualization kvm. [ 11.911060] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.419630] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.423555] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.428598] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.434280] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.437938] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.446467] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.454720] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 12.592800] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.858497] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.159664] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.194861] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.301946] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.326960] EDAC sbridge: Ver: 1.1.2 [ 14.962345] Key type dns_resolver registered [ 15.279879] NFS: Registering the id_resolver key type [ 15.281966] Key type id_resolver registered [ 15.283797] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg643-server login: [ 41.070705] libcfs: loading out-of-tree module taints kernel. [ 41.091523] Key type ._llcrypt registered [ 41.093633] Key type .llcrypt registered [ 41.139548] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 53.474687] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 54.962269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.986623] alg: No test for adler32 (adler32-zlib) [ 56.397770] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 57.298467] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 59.103548] Key type lgssc registered [ 60.556777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 67.657240] hrtimer: interrupt took 3230839 ns [ 74.133623] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 118.204217] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 131.139868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 131.177426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 132.451818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 132.495200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 132.571149] Lustre: lustre-MDT0000: new disk, initializing [ 132.640927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 132.659175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 136.603609] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 149.430481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.518880] Lustre: 6494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 149.564251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 149.568874] Lustre: Skipped 1 previous similar message [ 149.650686] Lustre: lustre-MDT0001: new disk, initializing [ 149.714837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.749945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 149.762856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.891841] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.044041] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 167.199775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 167.439605] Lustre: lustre-OST0000: new disk, initializing [ 167.447057] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 167.455680] Lustre: 8428:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 167.522693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.080434] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.597496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 173.604061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 173.679473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 186.572790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.835653] Lustre: lustre-OST0001: new disk, initializing [ 186.845237] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 186.862790] Lustre: 9501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 186.946075] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.547940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 192.564398] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 192.643773] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 192.884570] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.180119] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 214.308291] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 219.945499] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing check_logdir /tmp/testlogs/ [ 223.920905] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing yml_node [ 227.953916] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 230.347823] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 233.019814] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 234.247850] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Fri Aug 7 07:20:17 EDT 2026 [ 248.384133] Lustre: DEBUG MARKER: excepting tests: 18b 23b [ 257.449949] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 267.749166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 267.758956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 267.770339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.281531] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.282201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.298165] Lustre: Skipped 2 previous similar messages [ 269.304352] Lustre: Skipped 2 previous similar messages [ 270.624318] Lustre: server umount lustre-MDT0000 complete [ 278.436109] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101663 with bad export cookie 7082945715385215144 [ 278.441505] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 278.459873] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 278.761739] Lustre: server umount lustre-MDT0001 complete [ 295.903132] Lustre: 3624:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786101664/real 1786101664] req@ffff9fd3c88b1180 x1872863105746048/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786101680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 295.928960] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.149904] Lustre: server umount lustre-OST0000 complete [ 299.679986] Lustre: 3626:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786101668/real 1786101668] req@ffff9fd3c88b2d80 x1872863105746304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786101684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 301.055226] Lustre: 3625:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786101669/real 1786101669] req@ffff9fd3fe360000 x1872863105746560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786101685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 305.015186] Lustre: server umount lustre-OST0001 complete [ 318.600797] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 320.940961] Key type lgssc unregistered [ 321.220469] LNet: 14774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.224731] LNetError: 14774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.236848] LNet: Removed LNI 192.168.206.143@tcp [ 322.099374] Key type .llcrypt unregistered [ 322.104041] Key type ._llcrypt unregistered [ 342.656162] Key type ._llcrypt registered [ 342.659761] Key type .llcrypt registered [ 342.763310] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 356.848304] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 357.736015] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 357.810907] alg: No test for adler32 (adler32-zlib) [ 358.935321] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 359.260947] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 360.935779] Key type lgssc registered [ 362.238974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 411.279390] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 424.253254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 424.272371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.482358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 425.522921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 425.599661] Lustre: lustre-MDT0000: new disk, initializing [ 425.674277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.699144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 430.469501] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 442.668950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.771933] Lustre: 19226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 442.809605] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 442.815500] Lustre: Skipped 1 previous similar message [ 442.870407] Lustre: lustre-MDT0001: new disk, initializing [ 442.915029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 442.939933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 442.952090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 446.940527] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 451.358615] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 459.763681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 460.014844] Lustre: lustre-OST0000: new disk, initializing [ 460.019224] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 460.031627] Lustre: 21164:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 460.103068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.418338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 465.426794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 465.510500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 466.732355] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 479.502839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 479.602323] Lustre: lustre-OST0001: new disk, initializing [ 479.605672] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 479.614790] Lustre: 22187:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 479.682464] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 485.515750] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 489.494672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 489.502869] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 489.546580] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 496.140659] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 503.113876] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 510.711537] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 07:24:53 (1786101893) === [ 513.344527] Lustre: DEBUG MARKER: == sanity-lfsck test 0: Control LFSCK manually =========== 07:24:55 (1786101895) [ 513.623630] Lustre: 19235:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 513.629364] Lustre: 19235:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 513.636920] Lustre: 19235:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 513.650500] Lustre: 19235:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 513.663042] Lustre: 19235:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 513.679219] Lustre: 19235:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 514.191500] Lustre: 19235:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 514.199484] Lustre: 19235:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 8 previous similar messages [ 514.202767] Lustre: 19235:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 514.207062] Lustre: 19235:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 514.214246] Lustre: 19235:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 514.221401] Lustre: 19235:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 514.226978] Lustre: 19235:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 514.239215] Lustre: 19235:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 514.244710] Lustre: 19235:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 514.248716] Lustre: 19235:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 514.253584] Lustre: 19235:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 514.256962] Lustre: 19235:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 515.224840] Lustre: 23227:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 515.233137] Lustre: 23227:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 38 previous similar messages [ 515.244222] Lustre: 23227:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 515.252489] Lustre: 23227:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 515.259587] Lustre: 23227:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 515.274460] Lustre: 23227:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 515.294602] Lustre: 23227:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 515.307718] Lustre: 23227:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 515.322820] Lustre: 23227:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 515.334913] Lustre: 23227:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 515.345203] Lustre: 23227:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 515.365992] Lustre: 23227:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 517.242895] Lustre: 19233:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 517.251392] Lustre: 19233:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 74 previous similar messages [ 517.267864] Lustre: 19233:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 517.284230] Lustre: 19233:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 74 previous similar messages [ 517.291622] Lustre: 19233:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 517.302876] Lustre: 19233:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 74 previous similar messages [ 517.313442] Lustre: 19233:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 517.323854] Lustre: 19233:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 74 previous similar messages [ 517.333884] Lustre: 19233:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 517.343478] Lustre: 19233:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 74 previous similar messages [ 517.349708] Lustre: 19233:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 517.360854] Lustre: 19233:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 74 previous similar messages [ 523.096084] Lustre: *** cfs_fail_loc=1600, val=3*** [ 524.330906] Lustre: 23375:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0001: opcode 2: before 259 < left 274, rollback = 2 [ 524.332493] Lustre: 21153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 524.345049] Lustre: 23375:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 161 previous similar messages [ 524.345074] Lustre: 23375:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 524.345077] Lustre: 23375:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 160 previous similar messages [ 524.345089] Lustre: 23375:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 524.345092] Lustre: 23375:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 160 previous similar messages [ 524.345097] Lustre: 23375:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 524.345100] Lustre: 23375:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 160 previous similar messages [ 524.345105] Lustre: 23375:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 524.345108] Lustre: 23375:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 160 previous similar messages [ 524.441956] Lustre: 21153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 163 previous similar messages [ 527.592057] Lustre: *** cfs_fail_loc=1600, val=3*** [ 534.584264] Lustre: 23536:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 274, rollback = 2 [ 534.597301] Lustre: 21154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 534.603568] Lustre: 23536:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 42 previous similar messages [ 534.603593] Lustre: 23536:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 534.603597] Lustre: 23536:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 42 previous similar messages [ 534.603845] Lustre: 23536:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 534.603851] Lustre: 23536:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 42 previous similar messages [ 534.603857] Lustre: 23536:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 534.603860] Lustre: 23536:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 42 previous similar messages [ 534.603865] Lustre: 23536:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 534.603868] Lustre: 23536:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 42 previous similar messages [ 534.741563] Lustre: 21154:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 42 previous similar messages [ 545.248455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 545.255163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.272894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.760480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.775893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 547.809357] Lustre: server umount lustre-MDT0000 complete [ 550.855811] LustreError: 19220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101935 with bad export cookie 17874925911906970130 [ 550.857584] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.867390] LustreError: 19220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 550.888763] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.890579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 550.905097] Lustre: Skipped 3 previous similar messages [ 550.922517] Lustre: Skipped 3 previous similar messages [ 551.239309] Lustre: server umount lustre-MDT0001 complete [ 560.370839] Lustre: server umount lustre-OST0000 complete [ 563.286385] Lustre: server umount lustre-OST0001 complete [ 570.763355] Lustre: DEBUG MARKER: == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 07:25:53 (1786101953) [ 583.658790] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 593.333833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.784638] LustreError: 26241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.811119] LustreError: 26241:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 593.874670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 597.644560] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 599.009259] LustreError: 26242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.131592] LustreError: 26242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.352906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.680088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 609.396400] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 612.362809] Lustre: 27381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 618.091862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 619.361274] LustreError: 27734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.369270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:40 to 0x280000401:65) [ 623.579498] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 624.609052] LustreError: 27734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.801326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 630.915067] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 630.923092] Lustre: Skipped 1 previous similar message [ 635.628235] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 636.399627] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:65) [ 642.190869] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 645.243165] Lustre: 29250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 647.038867] Lustre: 29263:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 647.047830] Lustre: 29263:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 18 previous similar messages [ 647.054424] Lustre: 29263:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 647.066913] Lustre: 29263:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 16 previous similar messages [ 647.076320] Lustre: 29263:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 647.092910] Lustre: 29263:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 19 previous similar messages [ 647.109746] Lustre: 29263:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 647.120758] Lustre: 29263:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 19 previous similar messages [ 647.142971] Lustre: 29263:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 647.156612] Lustre: 29263:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 19 previous similar messages [ 647.174176] Lustre: 29263:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 647.185131] Lustre: 29263:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 19 previous similar messages [ 652.981081] Lustre: *** cfs_fail_loc=1501, val=0*** [ 659.855496] Lustre: Failing over lustre-MDT0000 [ 660.066822] Lustre: server umount lustre-MDT0000 complete [ 661.984523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 661.993613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 662.016561] LustreError: 28465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 662.031461] LustreError: 28465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 669.480627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 669.562926] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.838817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.899571] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 674.944604] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 675.298959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 675.299524] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 675.348719] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 675.378452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 675.383411] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 679.319385] Lustre: *** cfs_fail_loc=1505, val=0*** [ 689.381525] Lustre: DEBUG MARKER: == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 07:27:51 (1786102071) [ 691.463693] Lustre: 26236:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 691.470086] Lustre: 26236:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 322 previous similar messages [ 691.480771] Lustre: 26236:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 691.490388] Lustre: 26236:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 691.497987] Lustre: 26236:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 691.510603] Lustre: 26236:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 691.520363] Lustre: 26236:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 691.530615] Lustre: 26236:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 691.537488] Lustre: 26236:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 691.543556] Lustre: 26236:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 691.549712] Lustre: 26236:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 691.569339] Lustre: 26236:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 698.340188] Lustre: *** cfs_fail_loc=1502, val=0*** [ 707.571390] Lustre: Failing over lustre-MDT0000 [ 707.823888] Lustre: server umount lustre-MDT0000 complete [ 711.136802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 711.148658] LustreError: 26238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.158600] Lustre: Skipped 4 previous similar messages [ 711.161536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 711.185932] LustreError: 26238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 718.874328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 719.012357] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 719.227611] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 722.908684] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 724.457451] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 724.463760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 724.479770] Lustre: Skipped 3 previous similar messages [ 724.511256] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 724.553769] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:169 to 0x2c0000401:193) [ 724.553858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 725.802662] Lustre: *** cfs_fail_loc=1505, val=0*** [ 733.113527] Lustre: DEBUG MARKER: == sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 07:28:35 (1786102115) [ 739.481295] Lustre: *** cfs_fail_loc=1504, val=0*** [ 739.491937] Lustre: *** cfs_fail_loc=1504, val=0*** [ 739.498990] Lustre: Skipped 1 previous similar message [ 747.206251] Lustre: Failing over lustre-MDT0000 [ 747.430070] Lustre: server umount lustre-MDT0000 complete [ 750.056364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 750.063220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.067339] LustreError: 26238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 750.070567] Lustre: Skipped 3 previous similar messages [ 750.103229] LustreError: 26238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 757.473304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 757.667198] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 757.929693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 757.941853] Lustre: Skipped 1 previous similar message [ 757.992854] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 762.987367] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 763.361640] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 763.369952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 763.373267] Lustre: Skipped 3 previous similar messages [ 763.384120] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 763.422613] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:257) [ 763.423216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:233 to 0x280000401:257) [ 766.788319] Lustre: *** cfs_fail_loc=1505, val=0*** [ 774.364933] Lustre: DEBUG MARKER: == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 07:29:17 (1786102157) [ 775.479587] Lustre: 29263:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 775.486915] Lustre: 29263:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 644 previous similar messages [ 775.493370] Lustre: 29263:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 775.498337] Lustre: 29263:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 775.504387] Lustre: 29263:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 775.511674] Lustre: 29263:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 775.517534] Lustre: 29263:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 775.525161] Lustre: 29263:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 775.535456] Lustre: 29263:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 775.541506] Lustre: 29263:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 775.546453] Lustre: 29263:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 775.551227] Lustre: 29263:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 781.051464] Lustre: *** cfs_fail_loc=1603, val=0*** [ 788.849371] Lustre: Failing over lustre-MDT0000 [ 788.960453] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 788.965416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 788.973114] Lustre: Skipped 5 previous similar messages [ 788.979247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 789.000888] Lustre: Skipped 3 previous similar messages [ 789.149703] Lustre: server umount lustre-MDT0000 complete [ 799.330175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 799.451569] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 799.799159] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 804.840969] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 804.857806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 804.861262] Lustre: Skipped 3 previous similar messages [ 804.889792] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 804.925916] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:297 to 0x2c0000401:321) [ 804.927061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:297 to 0x280000401:321) [ 805.307531] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 814.435994] Lustre: DEBUG MARKER: == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 07:29:57 (1786102197) [ 820.715941] Lustre: *** cfs_fail_loc=1604, val=0*** [ 827.854609] Lustre: Failing over lustre-MDT0000 [ 828.031539] Lustre: server umount lustre-MDT0000 complete [ 830.439144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.440473] LustreError: 26237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.456127] Lustre: Skipped 2 previous similar messages [ 830.473204] LustreError: 26237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 837.636897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 837.766959] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 838.056946] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 842.429452] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 843.233081] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 843.243506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 843.253919] Lustre: Skipped 3 previous similar messages [ 843.313770] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 843.373880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:361 to 0x2c0000401:385) [ 843.375151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:361 to 0x280000401:385) [ 851.706740] Lustre: DEBUG MARKER: == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 07:30:34 (1786102234) [ 858.581433] Lustre: *** cfs_fail_loc=1605, val=0*** [ 866.917514] Lustre: Failing over lustre-MDT0000 [ 867.153127] Lustre: server umount lustre-MDT0000 complete [ 879.762748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 879.850783] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 880.181118] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 885.218739] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 885.227332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 885.248943] Lustre: Skipped 3 previous similar messages [ 885.268151] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 885.347166] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:425 to 0x2c0000401:449) [ 885.348455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:425 to 0x280000401:449) [ 885.444638] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 896.249311] Lustre: DEBUG MARKER: == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 07:31:18 (1786102278) [ 903.510327] Lustre: 26237:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 903.516941] Lustre: 26237:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1220 previous similar messages [ 903.532850] Lustre: 26237:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 903.540329] Lustre: 26237:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1220 previous similar messages [ 903.549050] Lustre: 26237:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 903.557825] Lustre: 26237:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1220 previous similar messages [ 903.562547] Lustre: 26237:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 903.568494] Lustre: 26237:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1220 previous similar messages [ 903.573776] Lustre: 26237:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 903.578686] Lustre: 26237:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1220 previous similar messages [ 903.584194] Lustre: 26237:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 903.590176] Lustre: 26237:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1220 previous similar messages [ 903.596347] Lustre: *** cfs_fail_loc=161d, val=0*** [ 911.605875] Lustre: Failing over lustre-MDT0000 [ 911.924571] Lustre: server umount lustre-MDT0000 complete [ 915.935680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 915.943468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.969285] Lustre: Skipped 7 previous similar messages [ 921.903684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 922.059373] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 922.278389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.284210] Lustre: Skipped 3 previous similar messages [ 922.312805] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 926.441323] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 927.718745] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 927.727219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 927.739361] Lustre: Skipped 3 previous similar messages [ 927.782642] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 927.844422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:489 to 0x280000401:513) [ 927.846326] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 935.839253] Lustre: DEBUG MARKER: == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 07:31:58 (1786102318) [ 938.242732] Lustre: *** cfs_fail_loc=1603, val=0*** [ 947.297286] Lustre: DEBUG MARKER: == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 07:32:10 (1786102330) [ 953.666663] Lustre: *** cfs_fail_loc=1603, val=0*** [ 954.510070] Lustre: *** cfs_fail_loc=1604, val=0*** [ 964.366653] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 07:32:27 (1786102347) [ 999.892654] Lustre: Failing over lustre-MDT0000 [ 1000.140872] Lustre: server umount lustre-MDT0000 complete [ 1004.520830] LustreError: 26236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.524487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1004.550821] LustreError: 26236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 1006.382571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1014.899623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1021.409483] Lustre: 16389:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786102389/real 1786102389] req@ffff9fd2cca40380 x1872863423047680/t0(0) o400->MGC192.168.206.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786102405 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1021.438073] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1026.050898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1026.082772] Lustre: lustre-MDT0000: reset Object Index mappings [ 1031.649531] LustreError: 16386:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fd3fed23100 x1872863423056512/t0(0) o250->MGC192.168.206.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1032.239853] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1037.286727] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1037.294908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1037.317163] Lustre: Skipped 3 previous similar messages [ 1037.350710] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1037.424854] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:609) [ 1037.425806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:609) [ 1039.214149] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1043.808808] LustreError: 42892:0:(lfsck_engine.c:1018:lfsck_master_engine()) lustre-MDT0000-osd: master engine fail to verify the .lustre/lost+found/, go ahead: rc = -115 [ 1043.836637] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1045.919593] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1045.924287] Lustre: Skipped 1 previous similar message [ 1052.729511] Lustre: Failing over lustre-MDT0000 [ 1053.311830] Lustre: server umount lustre-MDT0000 complete [ 1057.759710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1057.764554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.774811] Lustre: Skipped 5 previous similar messages [ 1065.261951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1069.535728] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1071.146206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:641) [ 1071.146645] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:641) [ 1073.519708] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1081.174888] Lustre: DEBUG MARKER: == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 07:34:24 (1786102464) [ 1083.757696] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1093.756051] Lustre: Failing over lustre-MDT0000 [ 1094.224339] Lustre: server umount lustre-MDT0000 complete [ 1100.076855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1112.287135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1113.061451] Lustre: 16387:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786102481/real 1786102481] req@ffff9fd3ebeaf100 x1872863423141760/t0(0) o400->MGC192.168.206.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786102497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1122.807665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1122.840476] Lustre: lustre-MDT0000: reset Object Index mappings [ 1137.633549] LustreError: 16386:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fd3fe625180 x1872863423154816/t0(0) o250->MGC192.168.206.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1137.989530] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1137.998896] Lustre: Skipped 1 previous similar message [ 1142.409387] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1143.275683] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1143.290005] Lustre: Skipped 1 previous similar message [ 1143.290852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1143.301907] Lustre: Skipped 7 previous similar messages [ 1143.344039] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1143.365773] Lustre: Skipped 1 previous similar message [ 1143.409463] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:705) [ 1143.422891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:705) [ 1146.428485] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1146.433574] Lustre: Skipped 2 previous similar messages [ 1158.943484] Lustre: Failing over lustre-MDT0000 [ 1159.160965] Lustre: server umount lustre-MDT0000 complete [ 1169.511378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1169.648487] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.656514] LustreError: Skipped 2 previous similar messages [ 1174.275607] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1175.082856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:737) [ 1175.083434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:737) [ 1177.647678] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1177.650034] Lustre: Skipped 84 previous similar messages [ 1184.313510] Lustre: DEBUG MARKER: == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 07:36:07 (1786102567) [ 1185.875651] Lustre: 29263:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1185.889924] Lustre: 29263:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1040 previous similar messages [ 1185.904908] Lustre: 29263:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1185.916996] Lustre: 29263:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1042 previous similar messages [ 1185.933331] Lustre: 29263:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1185.944514] Lustre: 29263:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1042 previous similar messages [ 1185.951688] Lustre: 29263:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1185.958387] Lustre: 29263:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1042 previous similar messages [ 1185.966509] Lustre: 29263:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1185.973405] Lustre: 29263:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1042 previous similar messages [ 1185.980035] Lustre: 29263:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1185.986038] Lustre: 29263:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1042 previous similar messages [ 1193.051786] Lustre: *** cfs_fail_loc=1600, val=1*** [ 1193.056117] Lustre: Skipped 7 previous similar messages [ 1210.714900] Lustre: DEBUG MARKER: == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 07:36:33 (1786102593) [ 1219.496936] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1219.498886] Lustre: Skipped 8 previous similar messages [ 1242.234670] Lustre: DEBUG MARKER: == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 07:37:05 (1786102625) [ 1252.464328] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1252.471040] Lustre: Skipped 11 previous similar messages [ 1258.048503] Lustre: Failing over lustre-MDT0000 [ 1258.413273] Lustre: server umount lustre-MDT0000 complete [ 1262.047685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1262.054428] LustreError: 26236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.060735] LustreError: Skipped 1 previous similar message [ 1262.082728] LustreError: 26236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 83 previous similar messages [ 1268.836596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1269.160586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.165486] Lustre: Skipped 4 previous similar messages [ 1269.201318] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1269.211188] Lustre: Skipped 1 previous similar message [ 1273.209887] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1274.336665] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1274.339063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1274.353552] Lustre: Skipped 1 previous similar message [ 1274.361845] Lustre: Skipped 7 previous similar messages [ 1274.423455] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1274.432648] Lustre: Skipped 1 previous similar message [ 1274.512235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:853 to 0x280000401:897) [ 1274.512599] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:854 to 0x2c0000401:897) [ 1283.310583] Lustre: DEBUG MARKER: == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 07:37:45 (1786102665) [ 1300.740553] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1319.637318] Lustre: 52858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1342.371773] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1346.157734] Lustre: 53996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1353.315532] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1353.320393] Lustre: Skipped 81 previous similar messages [ 1355.912122] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1356.959788] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1357.983268] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1360.037143] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1360.046710] Lustre: Skipped 1 previous similar message [ 1360.834524] Lustre: Failing over lustre-MDT0000 [ 1361.048908] Lustre: server umount lustre-MDT0000 complete [ 1361.376734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.394594] Lustre: Skipped 15 previous similar messages [ 1370.403068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1375.417922] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1376.327114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:947 to 0x280000401:993) [ 1376.329648] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:946 to 0x2c0000401:961) [ 1385.585600] Lustre: DEBUG MARKER: == sanity-lfsck test 8: LFSCK state machine ============== 07:39:28 (1786102768) [ 1391.585496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.065125] Lustre: server umount lustre-MDT0000 complete [ 1398.072948] LustreError: 50034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786102782 with bad export cookie 17874925911907184288 [ 1398.083671] LustreError: 50034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1398.484485] Lustre: server umount lustre-MDT0001 complete [ 1412.665428] Lustre: server umount lustre-OST0000 complete [ 1426.683335] Lustre: server umount lustre-OST0001 complete [ 1434.927540] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 1442.801350] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1488.642602] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1498.625513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.863886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1498.926156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1499.053418] Lustre: lustre-MDT0000: new disk, initializing [ 1499.180587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1504.128321] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1516.189494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1516.481861] Lustre: 59070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1516.583537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1516.596184] Lustre: Skipped 1 previous similar message [ 1516.794227] Lustre: lustre-MDT0001: new disk, initializing [ 1516.992537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1517.019836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1522.951412] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1528.796686] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1537.670292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1537.960204] Lustre: lustre-OST0000: new disk, initializing [ 1537.965760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1537.972232] Lustre: 60704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1540.008663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1540.019693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1540.131822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1545.634885] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1557.561935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1557.739980] Lustre: lustre-OST0001: new disk, initializing [ 1557.744349] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1557.752634] Lustre: 61573:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1558.967237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1558.984121] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1559.032056] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1563.848369] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1572.751684] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1575.897030] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1586.686594] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1587.580938] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1587.586270] Lustre: Skipped 19 previous similar messages [ 1591.154232] Lustre: *** cfs_fail_loc=1601, val=2*** [ 1591.160137] Lustre: Skipped 5 previous similar messages [ 1606.411597] Lustre: Failing over lustre-MDT0000 [ 1606.696228] Lustre: server umount lustre-MDT0000 complete [ 1609.188669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1609.197974] LustreError: Skipped 1 previous similar message [ 1615.396974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1615.489593] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1615.497733] LustreError: Skipped 3 previous similar messages [ 1615.704533] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1615.711877] Lustre: Skipped 1 previous similar message [ 1619.962903] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1620.962172] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1620.983217] Lustre: Skipped 1 previous similar message [ 1620.991835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1621.007452] Lustre: Skipped 7 previous similar messages [ 1621.038772] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1621.045068] Lustre: Skipped 1 previous similar message [ 1621.089409] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1621.097586] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:65) [ 1621.097802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:65) [ 1627.767640] Lustre: Failing over lustre-MDT0000 [ 1627.957394] Lustre: server umount lustre-MDT0000 complete [ 1636.227167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1640.573441] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1642.033942] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1642.038954] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:97) [ 1642.039216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:97) [ 1646.193521] Lustre: Failing over lustre-MDT0000 [ 1646.476519] Lustre: server umount lustre-MDT0000 complete [ 1654.356546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1659.434786] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1659.930881] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:129) [ 1659.938379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:129) [ 1664.989022] Lustre: *** cfs_fail_loc=1602, val=2*** [ 1678.496406] Lustre: DEBUG MARKER: == sanity-lfsck test 9a: LFSCK speed control (1) ========= 07:44:21 (1786103061) [ 1692.804660] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1711.483795] Lustre: 68527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1736.100214] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1739.279951] Lustre: 69663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1748.133619] Lustre: 59078:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1748.150269] Lustre: 59078:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 2759 previous similar messages [ 1748.167530] Lustre: 59078:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1748.179327] Lustre: 59078:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 2759 previous similar messages [ 1748.198403] Lustre: 59078:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1748.214049] Lustre: 59078:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 2759 previous similar messages [ 1748.234612] Lustre: 59078:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1748.253020] Lustre: 59078:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 2759 previous similar messages [ 1748.278582] Lustre: 59078:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1748.292301] Lustre: 59078:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 2759 previous similar messages [ 1748.314414] Lustre: 59078:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1748.330398] Lustre: 59078:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 2759 previous similar messages [ 1848.853272] Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) ========= 07:47:11 (1786103231) [ 1897.461506] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1897.471643] Lustre: Skipped 4 previous similar messages [ 1921.870555] Lustre: *** cfs_fail_loc=160c, val=0*** [ 1921.873518] Lustre: Skipped 7 previous similar messages [ 1959.421831] Lustre: DEBUG MARKER: == sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 07:49:01 (1786103341) [ 2005.772638] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2006.794591] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2006.796593] Lustre: Skipped 42 previous similar messages [ 2008.872678] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2008.879520] Lustre: Skipped 112 previous similar messages [ 2012.875194] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2012.878825] Lustre: Skipped 205 previous similar messages [ 2020.883311] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2020.890152] Lustre: Skipped 427 previous similar messages [ 2036.889450] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2036.897452] Lustre: Skipped 831 previous similar messages [ 2068.937201] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2068.940373] Lustre: Skipped 1746 previous similar messages [ 2073.238098] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2073.248840] Lustre: Skipped 2599 previous similar messages [ 2306.260213] Lustre: DEBUG MARKER: == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 07:54:49 (1786103689) [ 2388.869356] Lustre: 59079:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 320, rollback = 2 [ 2388.879144] Lustre: 59079:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 36830 previous similar messages [ 2388.884928] Lustre: 59079:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 1/4/0 [ 2388.894078] Lustre: 59079:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2388.910264] Lustre: 59079:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 5/5/0, xattr_set: 7/320/0 [ 2388.916417] Lustre: 59079:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2388.922224] Lustre: 59079:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 6/46/0, punch: 0/0/0, quota 1/3/0 [ 2388.927443] Lustre: 59079:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2388.933178] Lustre: 59079:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 2/5/0 [ 2388.938395] Lustre: 59079:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2388.951524] Lustre: 59079:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1 [ 2388.960541] Lustre: 59079:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2462.672617] Lustre: server umount lustre-MDT0000 complete [ 2463.711902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2463.714810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2463.722132] LustreError: Skipped 2 previous similar messages [ 2463.725107] LustreError: 70098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.725117] LustreError: 70098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 2463.787458] Lustre: Skipped 21 previous similar messages [ 2466.398779] LustreError: 59065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786103851 with bad export cookie 17874925911907203384 [ 2466.402566] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.414947] LustreError: 59065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2466.437806] LustreError: Skipped 2 previous similar messages [ 2466.770619] Lustre: server umount lustre-MDT0001 complete [ 2480.260076] Lustre: server umount lustre-OST0000 complete [ 2493.749572] Lustre: server umount lustre-OST0001 complete [ 2498.884862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 2506.045696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2521.567384] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2526.691081] LustreError: 74950:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.143@tcp: failed processing log, type 4: rc = -110 [ 2552.415221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2552.422690] Lustre: Skipped 8 previous similar messages [ 2557.627789] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2560.672021] Lustre: 75532:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2560.683036] Lustre: *** cfs_fail_loc=160e, val=3*** [ 2563.754820] Lustre: 75532:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2570.896878] Lustre: DEBUG MARKER: == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 07:59:13 (1786103953) [ 2582.361197] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 2590.539978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.834761] LustreError: 74975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.850014] LustreError: 74975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2590.914939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3272 to 0x280000401:3297) [ 2594.098330] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2599.714894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.613233] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2604.711680] Lustre: 78197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2615.230367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2620.012823] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2620.908492] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3207 to 0x2c0000401:3233) [ 2625.575969] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2628.339888] Lustre: 79696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2631.958092] Lustre: *** cfs_fail_loc=160d, val=0*** [ 2636.601466] Lustre: Failing over lustre-OST0000 [ 2636.679780] Lustre: server umount lustre-OST0000 complete [ 2643.088172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2643.237594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2643.244605] Lustre: Skipped 3 previous similar messages [ 2643.250172] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2643.257396] Lustre: Skipped 2 previous similar messages [ 2644.971187] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2644.985264] Lustre: Skipped 2 previous similar messages [ 2645.003618] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2645.003758] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2645.003767] Lustre: Skipped 11 previous similar messages [ 2645.004125] Lustre: *** cfs_fail_loc=215, val=0*** [ 2645.015513] Lustre: Skipped 2 previous similar messages [ 2645.018819] Lustre: Skipped 2 previous similar messages [ 2648.174888] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2650.081859] Lustre: *** cfs_fail_loc=215, val=0*** [ 2650.083833] Lustre: Skipped 1 previous similar message [ 2650.999701] Lustre: 81095:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2651.012737] Lustre: 81095:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2653.176473] Lustre: Failing over lustre-OST0000 [ 2653.260669] Lustre: server umount lustre-OST0000 complete [ 2659.889434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2661.363313] Lustre: *** cfs_fail_loc=215, val=0*** [ 2661.370929] Lustre: Skipped 2 previous similar messages [ 2664.654825] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2666.463938] Lustre: *** cfs_fail_loc=215, val=0*** [ 2666.473635] Lustre: Skipped 1 previous similar message [ 2670.561446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2670.567353] Lustre: Skipped 4 previous similar messages [ 2675.680782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2675.688324] Lustre: Skipped 3 previous similar messages [ 2680.803227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2680.811865] Lustre: Skipped 3 previous similar messages [ 2683.360610] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2683.492635] Lustre: server umount lustre-MDT0000 complete [ 2686.079351] LustreError: 74956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104070 with bad export cookie 17874925911908770299 [ 2686.101266] LustreError: 74956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2686.322240] Lustre: server umount lustre-MDT0001 complete [ 2698.311745] Lustre: server umount lustre-OST0000 complete [ 2711.828350] Lustre: server umount lustre-OST0001 complete [ 2719.310384] Lustre: DEBUG MARKER: == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 08:01:42 (1786104102) [ 2730.834404] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 2738.322811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.622344] LustreError: 84350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.636486] LustreError: 84350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 2742.385751] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2748.845687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.500490] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2755.125763] Lustre: 85491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2760.170924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2763.497133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3362 to 0x280000401:3393) [ 2765.366360] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2772.615298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2772.732745] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2772.741379] Lustre: Skipped 4 previous similar messages [ 2773.733784] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3207 to 0x2c0000401:3265) [ 2779.206123] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2786.736433] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2790.258768] Lustre: 87358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2819.521732] Lustre: DEBUG MARKER: == sanity-lfsck test 12b: auto detect Lustre device ====== 08:03:22 (1786104202) [ 2831.974197] Lustre: DEBUG MARKER: == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 08:03:35 (1786104215) [ 2833.092474] Lustre: *** cfs_fail_loc=160f, val=0*** [ 2841.671902] Lustre: DEBUG MARKER: == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 08:03:44 (1786104224) [ 2844.994865] Lustre: *** cfs_fail_loc=1610, val=0*** [ 2844.996881] Lustre: Skipped 7 previous similar messages [ 2891.748763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.762900] Lustre: Skipped 3 previous similar messages [ 2897.581353] Lustre: server umount lustre-MDT0000 complete [ 2901.129784] LustreError: 84332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104285 with bad export cookie 17874925911908778783 [ 2901.157298] LustreError: 84332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2901.495195] Lustre: server umount lustre-MDT0001 complete [ 2916.108991] Lustre: server umount lustre-OST0000 complete [ 2917.343083] Lustre: 16389:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786104286/real 1786104286] req@ffff9fd2c6737800 x1872863427544576/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786104302 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2919.469513] Lustre: server umount lustre-OST0001 complete [ 2931.196719] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 2940.607826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.353651] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2953.234703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.442176] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2960.329908] Lustre: 93228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2966.399271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2972.542658] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2978.876740] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:161) [ 2981.809431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2986.163491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3554 to 0x280000401:3585) [ 2986.170729] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3361) [ 2986.179718] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:161) [ 2988.784647] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2995.357898] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2998.438791] Lustre: 95099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3003.891819] Lustre: DEBUG MARKER: == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 08:06:26 (1786104386) [ 3006.109560] Lustre: 92084:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 3006.118063] Lustre: 92084:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1361 previous similar messages [ 3006.122309] Lustre: 92084:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 3006.126934] Lustre: 92084:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1361 previous similar messages [ 3006.133340] Lustre: 92084:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3006.136698] Lustre: 92084:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1361 previous similar messages [ 3006.149449] Lustre: 92084:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3006.156056] Lustre: 92084:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1361 previous similar messages [ 3006.170210] Lustre: 92084:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 3006.175377] Lustre: 92084:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1361 previous similar messages [ 3006.180981] Lustre: 92084:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3006.186341] Lustre: 92084:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1361 previous similar messages [ 3009.484443] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3009.488856] Lustre: Skipped 63 previous similar messages [ 3030.495688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3030.503099] LustreError: Skipped 2 previous similar messages [ 3030.510621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3030.530393] Lustre: Skipped 4 previous similar messages [ 3035.784158] Lustre: server umount lustre-MDT0000 complete [ 3037.668607] LustreError: 92088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.699253] LustreError: 92088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3039.131653] LustreError: 92068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104423 with bad export cookie 17874925911908807203 [ 3039.147989] LustreError: 92068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3039.574735] Lustre: server umount lustre-MDT0001 complete [ 3052.289644] Lustre: server umount lustre-OST0000 complete [ 3066.026925] Lustre: server umount lustre-OST0001 complete [ 3081.976422] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3091.675887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.122715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3092.128402] Lustre: Skipped 4 previous similar messages [ 3095.971272] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3103.508228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.769894] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3110.379775] Lustre: 99132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3116.377948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3117.677502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3682 to 0x280000401:3713) [ 3122.096566] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3128.294876] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:193) [ 3128.822789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3134.189879] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3134.439625] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:193) [ 3134.447187] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3393) [ 3140.968513] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3149.680478] Lustre: DEBUG MARKER: == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 08:08:52 (1786104532) [ 3153.082519] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3153.084521] Lustre: Skipped 63 previous similar messages [ 3153.350146] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3162.277364] Lustre: DEBUG MARKER: == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 08:09:05 (1786104545) [ 3164.239772] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3164.241758] Lustre: Skipped 1 previous similar message [ 3164.297478] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3164.300360] Lustre: Skipped 2 previous similar messages [ 3172.972090] Lustre: DEBUG MARKER: == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 08:09:15 (1786104555) [ 3174.143930] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_15c MDS newer than 2.7.55, LU-6475 [ 3175.791480] Lustre: DEBUG MARKER: == sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 08:09:18 (1786104558) [ 3181.392049] Lustre: *** cfs_fail_loc=1709, val=0*** [ 3181.593456] LustreError: 98001:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200003ab2:0x1:0x0]: rc = -5 [ 3181.607777] LustreError: 98001:0:(mdt_reint.c:2644:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240002340:0x6c:0x0]/s61 failed: rc = -5 [ 3186.283812] LustreError: 101014:0:(lod_object.c:945:lod_load_lmv_shards()) lustre-MDT0001-mdtlov: the shard [0x200003ab1:0x45:0x0]:1 for the striped directory [0x240002340:0x8c:0x0] is out of the known LMV EA range [0 - 0], failout [ 3189.023199] LustreError: 97988:0:(lod_object.c:945:lod_load_lmv_shards()) lustre-MDT0001-mdtlov: the shard [0x200003ab1:0x45:0x0]:1 for the striped directory [0x240002340:0x8c:0x0] is out of the known LMV EA range [0 - 0], failout [ 3189.041277] LustreError: 97988:0:(mdt_handler.c:1498:mdt_getattr_internal()) lustre-MDT0001: getattr error for [0x240002340:0x8c:0x0]: rc = -5 [ 3226.592425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.594234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3226.603038] Lustre: Skipped 20 previous similar messages [ 3226.612762] Lustre: Skipped 7 previous similar messages [ 3231.545168] Lustre: server umount lustre-MDT0000 complete [ 3237.651184] LustreError: 97972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104622 with bad export cookie 17874925911908821938 [ 3237.652549] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3237.661823] LustreError: 97972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3237.688404] LustreError: Skipped 3 previous similar messages [ 3237.845673] Lustre: server umount lustre-MDT0001 complete [ 3254.766936] Lustre: server umount lustre-OST0000 complete [ 3257.823172] Lustre: 16388:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786104626/real 1786104626] req@ffff9fd2c753f480 x1872863427884800/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786104642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3258.847117] Lustre: 16389:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786104627/real 1786104627] req@ffff9fd2d0f02a00 x1872863427885056/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786104643 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3261.111409] Lustre: server umount lustre-OST0001 complete [ 3274.309810] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 3276.589580] Key type lgssc unregistered [ 3276.923848] LNet: 104832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3276.932401] LNetError: 104832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3276.959742] LNet: Removed LNI 192.168.206.143@tcp [ 3277.721148] Key type .llcrypt unregistered [ 3277.724054] Key type ._llcrypt unregistered [ 3295.965333] Key type ._llcrypt registered [ 3295.968761] Key type .llcrypt registered [ 3296.057170] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 3306.814850] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3308.064146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3308.077826] alg: No test for adler32 (adler32-zlib) [ 3309.038411] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 3309.230940] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 3310.871174] Key type lgssc registered [ 3311.891655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3347.377221] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3356.328528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3356.343570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.487540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3357.522743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3357.603276] Lustre: lustre-MDT0000: new disk, initializing [ 3357.668201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3357.696363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3360.827716] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3370.899191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.972757] Lustre: 109261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3371.000536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3371.007027] Lustre: Skipped 1 previous similar message [ 3371.077847] Lustre: lustre-MDT0001: new disk, initializing [ 3371.130891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3371.163247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3371.175879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3374.579808] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3378.625777] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3386.365231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3386.561137] Lustre: lustre-OST0000: new disk, initializing [ 3386.566341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3386.574424] Lustre: 111197:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3386.620655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3389.792118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3389.805498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3389.894316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3391.552869] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3403.038398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3403.186277] Lustre: lustre-OST0001: new disk, initializing [ 3403.189618] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3403.198804] Lustre: 112220:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3403.265957] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3408.631257] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3408.940431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3408.951687] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3408.998933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3418.844618] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3428.406231] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3433.443202] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 08:13:36 (1786104816) === [ 3439.143904] Lustre: DEBUG MARKER: == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 08:13:41 (1786104821) [ 3439.338166] Lustre: 109268:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 3439.352089] Lustre: 109268:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3439.361026] Lustre: 109268:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3439.366979] Lustre: 109268:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 3439.372639] Lustre: 109268:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 3439.379650] Lustre: 109268:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3439.864769] Lustre: 109266:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 3439.884573] Lustre: 109266:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 9 previous similar messages [ 3439.897957] Lustre: 109266:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3439.907338] Lustre: 109266:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3439.916485] Lustre: 109266:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3439.921462] Lustre: 109266:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3439.930853] Lustre: 109266:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 7/369/2 [ 3439.941915] Lustre: 109266:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3439.951324] Lustre: 109266:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 3439.956460] Lustre: 109266:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3439.970428] Lustre: 109266:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3439.983172] Lustre: 109266:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 3440.873410] Lustre: 112228:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 3440.892968] Lustre: 112228:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 152 previous similar messages [ 3440.898490] Lustre: 112228:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 3440.903154] Lustre: 112228:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 153 previous similar messages [ 3440.936679] Lustre: 109268:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3440.954973] Lustre: 109268:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 155 previous similar messages [ 3440.962992] Lustre: 109268:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 7/369/0 [ 3440.970430] Lustre: 109268:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 155 previous similar messages [ 3440.975939] Lustre: 109268:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 3440.986136] Lustre: 109268:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 155 previous similar messages [ 3440.993897] Lustre: 109268:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3441.002210] Lustre: 109268:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 155 previous similar messages [ 3442.340096] Lustre: *** cfs_fail_loc=1613, val=0*** [ 3451.094864] Lustre: DEBUG MARKER: == sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 08:13:53 (1786104833) [ 3452.410538] Lustre: 112228:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 278, rollback = 2 [ 3452.425202] Lustre: 112228:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 146 previous similar messages [ 3452.433227] Lustre: 112228:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 3452.439206] Lustre: 112228:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 145 previous similar messages [ 3452.452948] Lustre: 112228:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3452.466271] Lustre: 112228:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 143 previous similar messages [ 3452.473919] Lustre: 112228:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3452.479943] Lustre: 112228:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 143 previous similar messages [ 3452.486435] Lustre: 112228:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 3452.491476] Lustre: 112228:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 143 previous similar messages [ 3452.496873] Lustre: 112228:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3452.501433] Lustre: 112228:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 143 previous similar messages [ 3453.546876] Lustre: *** cfs_fail_loc=1614, val=0*** [ 3454.059040] Lustre: *** cfs_fail_loc=1614, val=103*** [ 3454.060935] Lustre: Skipped 1 previous similar message [ 3457.720591] Lustre: 111188:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 275, rollback = 2 [ 3457.731670] Lustre: 111188:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 11 previous similar messages [ 3457.744911] Lustre: 111188:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3457.754625] Lustre: 111188:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3457.764257] Lustre: 111188:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 3/275/0 [ 3457.774390] Lustre: 111188:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3457.784231] Lustre: 111188:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 1/4/0, quota 7/225/0 [ 3457.797251] Lustre: 111188:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3457.807778] Lustre: 111188:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 3457.816830] Lustre: 111188:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3457.823489] Lustre: 111188:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3457.828266] Lustre: 111188:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 3463.815564] Lustre: DEBUG MARKER: == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 08:14:06 (1786104846) [ 3466.171952] Lustre: *** cfs_fail_loc=1615, val=0*** [ 3466.174210] Lustre: Skipped 1 previous similar message [ 3481.912504] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_18b skipping excluded test 18b [ 3483.150305] Lustre: DEBUG MARKER: == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 08:14:26 (1786104866) [ 3483.475351] Lustre: 112228:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 3483.482020] Lustre: 112228:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 23 previous similar messages [ 3483.486961] Lustre: 112228:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 3483.497445] Lustre: 112228:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 3483.502301] Lustre: 112228:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3483.510386] Lustre: 112228:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 3483.516233] Lustre: 112228:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3483.523976] Lustre: 112228:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 3483.530184] Lustre: 112228:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 3483.535452] Lustre: 112228:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 3483.539967] Lustre: 112228:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3483.545269] Lustre: 112228:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 3485.393596] Lustre: *** cfs_fail_loc=1617, val=0*** [ 3485.451659] Lustre: *** cfs_fail_loc=1617, val=0*** [ 3487.227503] Lustre: *** cfs_fail_loc=1616, val=0*** [ 3487.237294] Lustre: Skipped 5 previous similar messages [ 3503.956688] Lustre: DEBUG MARKER: == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 08:14:46 (1786104886) [ 3504.293105] Lustre: 112228:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 3504.307603] Lustre: 112228:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 24 previous similar messages [ 3504.313650] Lustre: 112228:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 3504.318902] Lustre: 112228:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 3504.324038] Lustre: 112228:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3504.330878] Lustre: 112228:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 3504.335921] Lustre: 112228:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3504.342598] Lustre: 112228:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 3504.348613] Lustre: 112228:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 3504.354267] Lustre: 112228:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 3504.359613] Lustre: 112228:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3504.363163] Lustre: 112228:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 3505.937570] Lustre: *** cfs_fail_loc=1618, val=0*** [ 3505.944378] Lustre: Skipped 5 previous similar messages [ 3540.448423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3540.456115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3540.471567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3541.997133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3541.998517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.003546] Lustre: Skipped 1 previous similar message [ 3542.012921] Lustre: Skipped 1 previous similar message [ 3545.006341] Lustre: server umount lustre-MDT0000 complete [ 3547.984586] LustreError: 113045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104932 with bad export cookie 13092439927736514852 [ 3547.987951] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3547.996582] LustreError: 113045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3548.233373] Lustre: server umount lustre-MDT0001 complete [ 3561.498147] Lustre: server umount lustre-OST0000 complete [ 3573.454428] Lustre: server umount lustre-OST0001 complete [ 3586.090661] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3594.547053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3594.868459] LustreError: 117934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3594.884521] LustreError: 117934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3594.926596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3598.851681] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3600.359214] LustreError: 117935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3604.449880] LustreError: 117934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3605.761919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.046845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3610.112505] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3612.689949] Lustre: 119073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3617.968188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3619.175839] LustreError: 119425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3619.189198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:116 to 0x280000401:161) [ 3623.380822] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3624.428448] LustreError: 119427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3630.563970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.569764] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:33) [ 3630.730169] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3630.734996] Lustre: Skipped 1 previous similar message [ 3635.474180] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3636.215990] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 3636.225609] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:33) [ 3641.557624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3644.133962] Lustre: 120941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3655.154647] Lustre: DEBUG MARKER: == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 08:17:18 (1786105038) [ 3655.998813] Lustre: 117929:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 3656.004864] Lustre: 117929:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 21 previous similar messages [ 3656.010328] Lustre: 117929:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3656.014478] Lustre: 117929:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3656.019712] Lustre: 117929:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3656.024511] Lustre: 117929:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3656.029736] Lustre: 117929:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 3656.036068] Lustre: 117929:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3656.041569] Lustre: 117929:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 3656.046533] Lustre: 117929:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3656.050774] Lustre: 117929:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3656.055474] Lustre: 117929:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3657.720607] Lustre: *** cfs_fail_loc=1618, val=0*** [ 3657.725385] Lustre: Skipped 3 previous similar messages [ 3692.512776] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.515054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3692.528257] Lustre: Skipped 4 previous similar messages [ 3692.540804] Lustre: Skipped 3 previous similar messages [ 3696.366182] Lustre: server umount lustre-MDT0000 complete [ 3697.635304] LustreError: 120636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3697.662430] LustreError: 120636:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3699.632061] LustreError: 120254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786105084 with bad export cookie 13092439927736530091 [ 3699.633655] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3699.644443] LustreError: 120254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3699.852288] Lustre: server umount lustre-MDT0001 complete [ 3713.605692] Lustre: server umount lustre-OST0000 complete [ 3727.198598] Lustre: server umount lustre-OST0001 complete [ 3740.046501] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3748.928194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.320620] LustreError: 123507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3749.406571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3753.279143] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3761.347174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3766.107978] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3768.998750] Lustre: 124649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3775.359152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3780.238857] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3786.293491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3787.503419] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:65) [ 3787.523655] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 3790.928883] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3794.664392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 3794.668872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:65) [ 3802.359420] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3805.015591] Lustre: 126518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3809.443580] Lustre: 123507:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 252 < left 260, rollback = 2 [ 3809.450895] Lustre: 123507:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 21 previous similar messages [ 3809.455354] Lustre: 123507:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3809.458540] Lustre: 123507:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3809.463492] Lustre: 123507:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/260/0 [ 3809.469330] Lustre: 123507:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3809.475744] Lustre: 123507:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 0/0/0, quota 4/150/2 [ 3809.485209] Lustre: 123507:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3809.493846] Lustre: 123507:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 1/17/2, delete: 0/0/0 [ 3809.497948] Lustre: 123507:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3809.504067] Lustre: 123507:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3809.515118] Lustre: 123507:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 3809.545182] Lustre: *** cfs_fail_loc=1602, val=10*** [ 3830.479166] Lustre: DEBUG MARKER: == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 08:20:13 (1786105213) [ 3832.531187] Lustre: *** cfs_fail_loc=1616, val=0*** [ 3832.535694] Lustre: Skipped 3 previous similar messages [ 3837.715879] Lustre: *** cfs_fail_loc=161c, val=0*** [ 3837.720149] Lustre: Skipped 1 previous similar message [ 3852.354690] Lustre: DEBUG MARKER: == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) ========================================================== 08:20:35 (1786105235) [ 3854.303585] Lustre: *** cfs_fail_loc=162e, val=0*** [ 3854.311353] Lustre: *** cfs_fail_loc=162e, val=0*** [ 3854.315800] Lustre: Skipped 7 previous similar messages [ 3867.062967] Lustre: DEBUG MARKER: == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 08:20:49 (1786105249) [ 3886.265255] Lustre: DEBUG MARKER: == sanity-lfsck test 19a: OST-object inconsistency self detect ========================================================== 08:21:09 (1786105269) [ 3897.897674] Lustre: DEBUG MARKER: == sanity-lfsck test 19b: OST-object inconsistency self repair ========================================================== 08:21:21 (1786105281) [ 3900.753650] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3900.775067] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3900.778361] Lustre: Skipped 3 previous similar messages [ 3905.194599] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.206.43@tcp inode [0x2000013a1:0x1f:0x0] object 0x280000401:203 extent [0-4095], client returned csum 0 (type 4), server csum 703fb3ad (type 4) [ 3906.090815] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.206.43@tcp inode [0x2000013a1:0x20:0x0] object 0x280000401:204 extent [0-4095], client returned csum 0 (type 4), server csum 7cbb935c (type 4) [ 3913.634501] Lustre: DEBUG MARKER: == sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================================================== 08:21:36 (1786105296) [ 3916.159861] Lustre: *** cfs_fail_loc=1616, val=0*** [ 3916.171536] Lustre: Skipped 3 previous similar messages [ 3937.417437] Lustre: DEBUG MARKER: == sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case ========================================================== 08:22:00 (1786105320) [ 3943.656947] Lustre: DEBUG MARKER: == sanity-lfsck test 21: run all LFSCK components by default ========================================================== 08:22:06 (1786105326) [ 3944.074734] Lustre: 126982:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 3944.090587] Lustre: 126982:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 135 previous similar messages [ 3944.104807] Lustre: 126982:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3944.113619] Lustre: 126982:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 135 previous similar messages [ 3944.119902] Lustre: 126982:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3944.124509] Lustre: 126982:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 135 previous similar messages [ 3944.135652] Lustre: 126982:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 3944.145167] Lustre: 126982:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 135 previous similar messages [ 3944.150844] Lustre: 126982:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 3944.156610] Lustre: 126982:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 135 previous similar messages [ 3944.162330] Lustre: 126982:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3944.167770] Lustre: 126982:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 135 previous similar messages [ 3953.304323] Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) ========================================================== 08:22:16 (1786105336) [ 3955.325433] Lustre: *** cfs_fail_loc=161e, val=0*** [ 3955.333812] Lustre: *** cfs_fail_loc=161e, val=0*** [ 3955.336607] Lustre: Skipped 1 previous similar message [ 3963.553610] Lustre: DEBUG MARKER: == sanity-lfsck test 22b: LFSCK can repair unmatched pairs (2) ========================================================== 08:22:26 (1786105346) [ 3964.968345] Lustre: *** cfs_fail_loc=161e, val=0*** [ 3964.971281] Lustre: *** cfs_fail_loc=161e, val=0*** [ 3974.613675] Lustre: DEBUG MARKER: == sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) ========================================================== 08:22:37 (1786105357) [ 3975.822905] Lustre: *** cfs_fail_loc=1620, val=0*** [ 3985.542541] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_23b skipping excluded test 23b [ 3986.728989] Lustre: DEBUG MARKER: == sanity-lfsck test 23c: LFSCK can repair dangling name entry (3) ========================================================== 08:22:49 (1786105369) [ 3990.696493] Lustre: *** cfs_fail_loc=1621, val=127*** [ 3990.702631] Lustre: Skipped 1 previous similar message [ 3992.752315] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4010.883843] Lustre: DEBUG MARKER: == sanity-lfsck test 23d: LFSCK can repair a dangling name entry to a remote object ========================================================== 08:23:13 (1786105393) [ 4012.603940] Lustre: Failing over lustre-MDT0000 [ 4012.866085] Lustre: server umount lustre-MDT0000 complete [ 4015.073123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.075156] LustreError: 123502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4015.087956] Lustre: Skipped 2 previous similar messages [ 4015.113875] LustreError: 123502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4020.217333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4020.343357] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4020.497665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4020.507127] Lustre: Skipped 3 previous similar messages [ 4020.545700] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4023.974127] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4025.394550] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4025.831274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4025.856171] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4025.897039] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:130 to 0x2c0000401:161) [ 4025.897123] LustreError: 125767:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0000: [0x2000013a3:0x82:0x0] doesn't exist!: rc = -14 [ 4025.897974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 4035.018722] Lustre: DEBUG MARKER: == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ========================================================== 08:23:37 (1786105417) [ 4036.937307] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4037.053570] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4037.057562] Lustre: Skipped 1 previous similar message [ 4046.026446] Lustre: DEBUG MARKER: == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ========================================================== 08:23:48 (1786105428) [ 4047.349727] Lustre: *** cfs_fail_loc=1623, val=0*** [ 4055.327952] Lustre: DEBUG MARKER: == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace ========================================================== 08:23:58 (1786105438) [ 4056.647456] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4065.122461] Lustre: DEBUG MARKER: == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========================================================== 08:24:08 (1786105448) [ 4075.439458] Lustre: DEBUG MARKER: == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan ========================================================== 08:24:18 (1786105458) [ 4076.829087] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4076.832199] Lustre: Skipped 1 previous similar message [ 4085.917223] Lustre: DEBUG MARKER: == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ========================================================== 08:24:28 (1786105468) [ 4094.954948] Lustre: DEBUG MARKER: == sanity-lfsck test 28: Skip the failed MDT(s) when handle orphan MDT-objects ========================================================== 08:24:38 (1786105478) [ 4099.961902] Lustre: *** cfs_fail_loc=161c, val=0*** [ 4099.970424] Lustre: Skipped 1 previous similar message [ 4113.175860] Lustre: DEBUG MARKER: == sanity-lfsck test 29b: LFSCK can repair bad nlink count (2) ========================================================== 08:24:56 (1786105496) [ 4114.927528] Lustre: *** cfs_fail_loc=1626, val=0*** [ 4114.931137] Lustre: Skipped 4 previous similar messages [ 4123.672899] Lustre: DEBUG MARKER: == sanity-lfsck test 29c: verify linkEA size limitation == 08:25:06 (1786105506) [ 4149.813940] Lustre: DEBUG MARKER: == sanity-lfsck test 29d: accessing non-existing inode shouldn't turn fs read-only (ldiskfs) ========================================================== 08:25:32 (1786105532) [ 4152.907102] LustreError: 123501:0:(osd_handler.c:284:osd_idc_find_or_init()) lustre-MDT0000: cannot lookup FID [0x2000013a3:0x9e:0x0]: rc = -2 [ 4158.988189] Lustre: DEBUG MARKER: == sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 08:25:41 (1786105541) [ 4191.886644] Lustre: Failing over lustre-MDT0000 [ 4192.221942] Lustre: server umount lustre-MDT0000 complete [ 4194.784050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.784311] LustreError: 123502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.784904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4194.793485] Lustre: Skipped 3 previous similar messages [ 4194.829341] LustreError: 123502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4200.931780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4201.026086] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4201.248568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4201.289608] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4205.241889] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4206.564935] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4206.595355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4206.610122] Lustre: Skipped 3 previous similar messages [ 4206.637507] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4206.702824] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:167 to 0x2c0000401:193) [ 4206.706052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:294 to 0x280000401:321) [ 4207.826186] Lustre: 142237:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 264, rollback = 2 [ 4207.836803] Lustre: 142237:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 773 previous similar messages [ 4207.848315] Lustre: 142237:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4207.853129] Lustre: 142237:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 773 previous similar messages [ 4207.860717] Lustre: 142237:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/264/0 [ 4207.870619] Lustre: 142237:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 773 previous similar messages [ 4207.879125] Lustre: 142237:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4207.890423] Lustre: 142237:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 773 previous similar messages [ 4207.900543] Lustre: 142237:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/32/1, delete: 1/1/0 [ 4207.912599] Lustre: 142237:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 773 previous similar messages [ 4207.931189] Lustre: 142237:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4207.939311] Lustre: 142237:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 773 previous similar messages [ 4215.449116] Lustre: DEBUG MARKER: == sanity-lfsck test 31a: The LFSCK can find/repair the name entry with bad name hash (1) ========================================================== 08:26:38 (1786105598) [ 4227.173978] Lustre: DEBUG MARKER: == sanity-lfsck test 31b: The LFSCK can find/repair the name entry with bad name hash (2) ========================================================== 08:26:50 (1786105610) [ 4241.665412] Lustre: DEBUG MARKER: == sanity-lfsck test 31c: Re-generate the lost master LMV EA for striped directory ========================================================== 08:27:04 (1786105624) [ 4243.211697] Lustre: *** cfs_fail_loc=1629, val=0*** [ 4243.227980] Lustre: Skipped 5 previous similar messages [ 4273.549371] Lustre: DEBUG MARKER: == sanity-lfsck test 31d: Set broken striped directory (modified after broken) as read-only ========================================================== 08:27:36 (1786105656) [ 4279.407494] Lustre: Failing over lustre-MDT0000 [ 4279.688281] Lustre: server umount lustre-MDT0000 complete [ 4283.363839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4283.364776] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.392741] Lustre: Skipped 3 previous similar messages [ 4286.907681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.033886] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4287.277087] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4291.321483] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4292.576714] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4292.614514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4292.628421] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4292.632075] Lustre: Skipped 3 previous similar messages [ 4292.692323] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:167 to 0x2c0000401:225) [ 4292.700065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:294 to 0x280000401:353) [ 4300.274961] Lustre: Failing over lustre-MDT0000 [ 4300.604576] Lustre: server umount lustre-MDT0000 complete [ 4302.816918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4302.838465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4302.848526] Lustre: Skipped 2 previous similar messages [ 4308.075365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4308.195132] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4308.465223] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4309.054366] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4312.071570] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4313.580801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4313.597460] Lustre: Skipped 3 previous similar messages [ 4313.641570] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4313.690198] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:167 to 0x2c0000401:257) [ 4313.693546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4319.026174] Lustre: DEBUG MARKER: == sanity-lfsck test 31e: Re-generate the lost slave LMV EA for striped directory (1) ========================================================== 08:28:22 (1786105702) [ 4328.768646] Lustre: DEBUG MARKER: == sanity-lfsck test 31f: Re-generate the lost slave LMV EA for striped directory (2) ========================================================== 08:28:31 (1786105711) [ 4338.199950] Lustre: DEBUG MARKER: == sanity-lfsck test 31g: Repair the corrupted slave LMV EA ========================================================== 08:28:41 (1786105721) [ 4380.545987] Lustre: DEBUG MARKER: == sanity-lfsck test 31h: Repair the corrupted shard's name entry ========================================================== 08:29:23 (1786105763) [ 4382.706482] Lustre: *** cfs_fail_loc=162c, val=0*** [ 4382.715170] Lustre: Skipped 13 previous similar messages [ 4396.903672] Lustre: DEBUG MARKER: == sanity-lfsck test 32a: stop LFSCK when some OST failed ========================================================== 08:29:39 (1786105779) [ 4404.685754] LustreError: 148019:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 4406.840951] Lustre: Failing over lustre-OST0000 [ 4406.930836] Lustre: server umount lustre-OST0000 complete [ 4407.264711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4407.276053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.289258] Lustre: Skipped 2 previous similar messages [ 4407.299518] LustreError: 132264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.313415] LustreError: 132264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 4407.751104] LustreError: 148019:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 4407.754928] LustreError: 148019:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 4409.577958] LustreError: 148019:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout interrupted [ 4420.497746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4420.698438] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4422.061356] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4422.090410] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4422.090894] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4422.115980] Lustre: Skipped 3 previous similar messages [ 4426.180224] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4433.537297] Lustre: DEBUG MARKER: == sanity-lfsck test 32b: stop LFSCK when some MDT failed ========================================================== 08:30:16 (1786105816) [ 4443.803219] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4459.492295] Lustre: 150820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4478.793928] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4481.811605] Lustre: 151962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4491.587895] LustreError: 152081:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d sleeping for 3000ms [ 4493.976850] Lustre: Failing over lustre-MDT0001 [ 4494.631231] LustreError: 152081:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 4494.649101] LustreError: 152080:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d sleeping for 3000ms [ 4494.653928] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 4494.668089] LustreError: 152080:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) Skipped 1 previous similar message [ 4494.713065] LustreError: Skipped 1 previous similar message [ 4494.715157] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.732348] Lustre: Skipped 1 previous similar message [ 4494.739403] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4494.933339] Lustre: server umount lustre-MDT0001 complete [ 4497.287566] LustreError: 152080:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout interrupted [ 4506.996921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4507.222806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4507.230219] Lustre: Skipped 3 previous similar messages [ 4507.260090] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 4511.490407] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4512.233621] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4512.246654] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4512.255542] Lustre: Skipped 1 previous similar message [ 4512.302796] Lustre: lustre-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4512.357508] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:97) [ 4512.357833] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:97) [ 4519.331883] Lustre: DEBUG MARKER: == sanity-lfsck test 33: check LFSCK paramters =========== 08:31:42 (1786105902) [ 4532.213301] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4549.191604] Lustre: 154801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4567.077601] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4569.973838] Lustre: 155935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4590.737518] Lustre: DEBUG MARKER: == sanity-lfsck test 34: LFSCK can rebuild the lost agent object ========================================================== 08:32:53 (1786105973) [ 4592.198270] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_34 Only valid for ZFS backend [ 4593.850725] Lustre: DEBUG MARKER: == sanity-lfsck test 35: LFSCK can rebuild the lost agent entry ========================================================== 08:32:56 (1786105976) [ 4601.024929] Lustre: *** cfs_fail_loc=1631, val=0*** [ 4614.626589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4614.634380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4614.656338] Lustre: Skipped 3 previous similar messages [ 4619.640905] Lustre: server umount lustre-MDT0000 complete [ 4624.784952] LustreError: 151991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786106009 with bad export cookie 13092439927736603010 [ 4624.785551] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4624.804454] LustreError: 151991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4624.871370] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4624.885030] Lustre: Skipped 7 previous similar messages [ 4624.893534] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4624.917522] Lustre: Skipped 1 previous similar message [ 4625.148354] Lustre: server umount lustre-MDT0001 complete [ 4635.254329] Lustre: server umount lustre-OST0000 complete [ 4639.081590] Lustre: server umount lustre-OST0001 complete [ 4651.901832] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4660.254494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.675575] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4665.825042] LustreError: 158685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.837139] LustreError: 158685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 4673.621359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4678.815369] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4681.918718] Lustre: 159826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4688.761812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4696.227666] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4698.288049] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:129) [ 4705.293128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4706.484991] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:129) [ 4706.500214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:412 to 0x2c0000401:449) [ 4706.578252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:540 to 0x280000401:577) [ 4712.115669] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4719.467075] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4722.730491] Lustre: 161696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4731.362574] Lustre: DEBUG MARKER: == sanity-lfsck test 36a: rebuild LOV EA for mirrored file (1) ========================================================== 08:35:14 (1786106114) [ 4733.151467] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36a needs >= 3 OSTs [ 4735.185765] Lustre: DEBUG MARKER: == sanity-lfsck test 36b: rebuild LOV EA for mirrored file (2) ========================================================== 08:35:17 (1786106117) [ 4736.790584] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36b needs >= 3 OSTs [ 4738.471279] Lustre: DEBUG MARKER: == sanity-lfsck test 36c: rebuild LOV EA for mirrored file (3) ========================================================== 08:35:21 (1786106121) [ 4740.052575] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36c needs >= 3 OSTs [ 4741.752441] Lustre: DEBUG MARKER: == sanity-lfsck test 37: LFSCK must skip a ORPHAN ======== 08:35:24 (1786106124) [ 4743.834327] Lustre: 158681:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 4743.847348] Lustre: 158681:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1607 previous similar messages [ 4743.853648] Lustre: 158681:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 4743.862833] Lustre: 158681:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1607 previous similar messages [ 4743.870332] Lustre: 158681:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4743.877648] Lustre: 158681:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1607 previous similar messages [ 4743.882815] Lustre: 158681:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4743.889245] Lustre: 158681:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1607 previous similar messages [ 4743.895660] Lustre: 158681:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 4743.904511] Lustre: 158681:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1607 previous similar messages [ 4743.911947] Lustre: 158681:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4743.918075] Lustre: 158681:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1607 previous similar messages [ 4753.419705] Lustre: DEBUG MARKER: == sanity-lfsck test 38: LFSCK does not break foreign file and reverse is also true ========================================================== 08:35:36 (1786106136) [ 4766.771888] Lustre: DEBUG MARKER: == sanity-lfsck test 39: LFSCK does not break foreign dir and reverse is also true ========================================================== 08:35:49 (1786106149) [ 4782.400330] Lustre: DEBUG MARKER: == sanity-lfsck test 40a: LFSCK correctly fixes lmm_oi in composite layout ========================================================== 08:36:04 (1786106164) [ 4797.376302] Lustre: DEBUG MARKER: == sanity-lfsck test 41: SEL support in LFSCK ============ 08:36:19 (1786106179) [ 4814.225479] Lustre: DEBUG MARKER: == sanity-lfsck test 42: LFSCK repairs inconsistent MDT-object/OST-object encryption flags ========================================================== 08:36:37 (1786106197) [ 4849.013383] Lustre: *** cfs_fail_loc=1632, val=0*** [ 4862.367940] Lustre: DEBUG MARKER: == sanity-lfsck test 43: LFSCK does not loop endlessly on iget failure in scanning-phase1 ========================================================== 08:37:25 (1786106245) [ 4864.799372] Lustre: Failing over lustre-MDT0001 [ 4865.105306] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.43@tcp (stopping) [ 4870.187396] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.43@tcp (stopping) [ 4870.196955] Lustre: Skipped 3 previous similar messages [ 4871.323594] Lustre: server umount lustre-MDT0001 complete [ 4878.993270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.432344] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 4879.435320] Lustre: lustre-MDT0001: Aborting client recovery [ 4879.447546] LustreError: 165484:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0001: Aborting recovery [ 4879.455297] Lustre: 165508:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4879.464084] LustreError: 165507:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0000-osp-MDT0001: get update log duration 0, retries 0, failed: rc = -108 [ 4879.481067] Lustre: 165508:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 1032d558-1788-4e03-a6f3-997a224cbaf1@ [ 4879.487492] Lustre: lustre-MDT0001: disconnecting 2 stale clients [ 4879.492474] Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] [ 4879.510655] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] [ 4879.585795] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4879.588245] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:161) [ 4884.112506] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4884.452824] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4884.479863] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4884.487182] Lustre: Skipped 2 previous similar messages [ 4887.266170] Lustre: *** cfs_fail_loc=1e2, val=483*** [ 4891.343930] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid [ 4891.611265] Lustre: DEBUG MARKER: osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid in FULL state after 0 sec [ 4897.411391] Lustre: DEBUG MARKER: == sanity-lfsck test 44: umount while lfsck is stopping == 08:38:00 (1786106280) [ 4905.361299] Lustre: *** cfs_fail_loc=1600, val=3*** [ 4907.918613] Lustre: Failing over lustre-MDT0000 [ 4910.049622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4910.051290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4910.052105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4910.052110] Lustre: Skipped 3 previous similar messages [ 4910.065266] Lustre: Skipped 4 previous similar messages [ 4910.395816] Lustre: server umount lustre-MDT0000 complete [ 4922.061251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.211709] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4922.654980] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4922.679628] Lustre: Skipped 2 previous similar messages [ 4926.511543] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4927.993558] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4928.003759] Lustre: Skipped 1 previous similar message [ 4928.059191] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4928.142958] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 4928.145247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:621 to 0x280000401:641) [ 4928.858933] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4940.633297] Lustre: DEBUG MARKER: == sanity-lfsck test 45: LFSCK should fix UID/GID/PROJID of OST object ========================================================== 08:38:43 (1786106323) [ 4966.480164] Lustre: Failing over lustre-OST0000 [ 4966.660642] Lustre: server umount lustre-OST0000 complete [ 4972.777222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4982.605477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4982.829349] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4984.440868] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4988.082093] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4994.294863] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4994.550422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4999.939396] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5000.190963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5006.685679] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d40c591b000.ost_server_uuid 50 [ 5008.489199] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d40c591b000.ost_server_uuid in FULL state after 0 sec [ 5064.676373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5064.681030] Lustre: Skipped 3 previous similar messages [ 5068.532995] Lustre: server umount lustre-MDT0000 complete [ 5076.349375] LustreError: 158667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786106461 with bad export cookie 13092439927736685645 [ 5076.352324] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5076.365939] LustreError: 158667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5076.644857] Lustre: server umount lustre-MDT0001 complete [ 5094.557270] Lustre: server umount lustre-OST0000 complete [ 5095.392908] Lustre: 106428:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786106464/real 1786106464] req@ffff9fd3eb290e00 x1872866517589376/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786106480 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5096.415077] Lustre: 106425:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786106465/real 1786106465] req@ffff9fd2c8463800 x1872866517589760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786106481 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5100.583529] Lustre: 106428:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786106469/real 1786106469] req@ffff9fd2c8460000 x1872866517590016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786106485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5102.635101] Lustre: 106425:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786106471/real 1786106471] req@ffff9fd2c8462680 x1872866517590528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786106487 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5103.715795] Lustre: server umount lustre-OST0001 complete [ 5120.581483] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 5123.422531] Key type lgssc unregistered [ 5123.707806] LNet: 174621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5123.715668] LNetError: 174621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5123.731741] LNet: Removed LNI 192.168.206.143@tcp [ 5124.682111] Key type .llcrypt unregistered [ 5124.684714] Key type ._llcrypt unregistered [ 5145.645641] Key type ._llcrypt registered [ 5145.653028] Key type .llcrypt registered [ 5145.853697] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 5169.557500] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 5171.319552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5171.552207] alg: No test for adler32 (adler32-zlib) [ 5172.960607] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 5173.492687] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 5175.384070] Key type lgssc registered [ 5177.526396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5229.076565] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 5243.155959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5243.204131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5244.402831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5244.427793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5244.505990] Lustre: lustre-MDT0000: new disk, initializing [ 5244.573143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5244.588535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5249.843307] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5265.153828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5265.333450] Lustre: 179078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5265.387538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5265.394119] Lustre: Skipped 1 previous similar message [ 5265.571398] Lustre: lustre-MDT0001: new disk, initializing [ 5265.660693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5265.713795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5265.735595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5271.044960] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5276.345714] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5285.056610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5285.226525] Lustre: lustre-OST0000: new disk, initializing [ 5285.229410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5285.236729] Lustre: 181017:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5285.316780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5286.187836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5286.205337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5286.345024] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5291.453991] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5305.288102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5305.422866] Lustre: lustre-OST0001: new disk, initializing [ 5305.425990] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5305.430444] Lustre: 182046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5305.478579] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5311.280794] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5312.096093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5312.107443] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5312.236707] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5323.155569] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5329.620724] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5337.344678] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 08:45:19 (1786106719) === [ 5339.434159] Lustre: DEBUG MARKER: == sanity-lfsck test complete, duration 5103 sec ========= 08:45:21 (1786106721) [ 5341.031390] Lustre: DEBUG MARKER: === sanity-lfsck: start cleanup 08:45:23 (1786106723) === [ 5345.076883] Lustre: DEBUG MARKER: === sanity-lfsck: finish cleanup 08:45:27 (1786106727) === [ 5352.928209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5352.942797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.966223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5353.442472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5353.456378] Lustre: Skipped 1 previous similar message [ 5353.467425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5353.477399] Lustre: Skipped 3 previous similar messages [ 5355.486679] Lustre: server umount lustre-MDT0000 complete [ 5363.680886] LustreError: 179085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5363.697232] LustreError: 179085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5363.783403] LustreError: 182043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786106748 with bad export cookie 13012584468671736137 [ 5363.787134] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5363.804934] LustreError: 182043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5364.042538] Lustre: server umount lustre-MDT0001 complete [ 5382.403273] Lustre: server umount lustre-OST0000 complete [ 5384.673769] Lustre: 176240:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786106753/real 1786106753] req@ffff9fd3ec598380 x1872868470244736/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786106769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5384.739737] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5384.762336] Lustre: Skipped 1 previous similar message [ 5388.255102] Lustre: 176237:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786106757/real 1786106757] req@ffff9fd2cc0ee680 x1872868470244992/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786106773 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5389.285588] Lustre: 176239:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786106758/real 1786106758] req@ffff9fd2cc0edf80 x1872868470245376/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786106774 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5393.453513] Lustre: server umount lustre-OST0001 complete [ 5414.064608] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 5418.369775] Key type lgssc unregistered [ 5418.743884] LNet: 185520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5418.758300] LNetError: 185520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5418.793733] LNet: Removed LNI 192.168.206.143@tcp [ 5419.947516] Key type .llcrypt unregistered [ 5419.949369] Key type ._llcrypt unregistered