[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 537891201 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002378] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004026] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.012109] LSM: Security Framework initializing [ 0.013058] Yama: becoming mindful. [ 0.014058] SELinux: Initializing. [ 0.015095] *** VALIDATE selinux *** [ 0.024000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028352] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029174] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030402] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031127] *** VALIDATE tmpfs *** [ 0.032000] *** VALIDATE proc *** [ 0.032000] *** VALIDATE cgroup *** [ 0.032000] *** VALIDATE cgroup2 *** [ 0.032294] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034202] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035030] Spectre V2 : User space: Vulnerable [ 0.036013] Speculative Store Bypass: Vulnerable [ 0.040048] debug: unmapping init [mem 0xffffffffad259000-0xffffffffad260fff] [ 0.042201] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043767] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045031] ... bit width: 48 [ 0.046012] ... generic registers: 4 [ 0.047032] ... value mask: 0000ffffffffffff [ 0.048029] ... max period: 00007fffffffffff [ 0.049033] ... fixed-purpose events: 3 [ 0.051000] ... event mask: 000000070000000f [ 0.052391] rcu: Hierarchical SRCU implementation. [ 0.055993] smp: Bringing up secondary CPUs ... [ 0.057079] x86: Booting SMP configuration: [ 0.058023] .... node #0, CPUs: #1 #2 #3 [ 0.074574] smp: Brought up 1 node, 4 CPUs [ 0.076211] smpboot: Max logical packages: 1 [ 0.077030] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.125000] node 0 deferred pages initialised in 44ms [ 0.131248] devtmpfs: initialized [ 0.132339] x86/mm: Memory block size: 128MB [ 0.135334] gcov: version magic: 0x41383552 [ 0.138403] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139158] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.140571] pinctrl core: initialized pinctrl subsystem [ 0.141283] [ 0.142012] ************************************************************* [ 0.143018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144023] ** ** [ 0.145024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.146026] ** ** [ 0.147021] ** This means that this kernel is built to expose internal ** [ 0.148026] ** IOMMU data structures, which may compromise security on ** [ 0.149023] ** your system. ** [ 0.150028] ** ** [ 0.151024] ** If you see this message and you are not debugging the ** [ 0.152030] ** kernel, report this immediately to your vendor! ** [ 0.153024] ** ** [ 0.154020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155022] ************************************************************* [ 0.160287] NET: Registered protocol family 16 [ 0.164025] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167717] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176036] cpuidle: using governor menu [ 0.187048] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.195071] PCI: Using configuration type 1 for base access [ 0.201165] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216145] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217109] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223247] cryptd: max_cpu_qlen set to 1000 [ 0.226493] ACPI: Added _OSI(Module Device) [ 0.227000] ACPI: Added _OSI(Processor Device) [ 0.227000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229026] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244119] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.260000] ACPI: Interpreter enabled [ 0.261068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.263019] ACPI: Using IOAPIC for interrupt routing [ 0.268259] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.274402] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.287225] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.290122] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.294030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.299109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.307198] acpiphp: Slot [2] registered [ 0.311281] acpiphp: Slot [5] registered [ 0.314250] acpiphp: Slot [6] registered [ 0.319186] acpiphp: Slot [7] registered [ 0.320000] acpiphp: Slot [8] registered [ 0.321305] acpiphp: Slot [9] registered [ 0.324740] acpiphp: Slot [10] registered [ 0.327234] acpiphp: Slot [3] registered [ 0.330171] acpiphp: Slot [4] registered [ 0.334322] acpiphp: Slot [11] registered [ 0.336185] acpiphp: Slot [12] registered [ 0.341819] acpiphp: Slot [13] registered [ 0.345210] acpiphp: Slot [14] registered [ 0.346217] acpiphp: Slot [15] registered [ 0.350237] acpiphp: Slot [16] registered [ 0.353548] acpiphp: Slot [17] registered [ 0.357260] acpiphp: Slot [18] registered [ 0.361486] acpiphp: Slot [19] registered [ 0.362174] acpiphp: Slot [20] registered [ 0.364182] acpiphp: Slot [21] registered [ 0.369504] acpiphp: Slot [22] registered [ 0.372175] acpiphp: Slot [23] registered [ 0.375430] acpiphp: Slot [24] registered [ 0.378442] acpiphp: Slot [25] registered [ 0.381475] acpiphp: Slot [26] registered [ 0.385242] acpiphp: Slot [27] registered [ 0.387152] acpiphp: Slot [28] registered [ 0.389126] acpiphp: Slot [29] registered [ 0.390303] acpiphp: Slot [30] registered [ 0.393280] acpiphp: Slot [31] registered [ 0.397133] PCI host bridge to bus 0000:00 [ 0.403045] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.409037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.411311] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.419034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.429051] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.433037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.434461] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.442222] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.459534] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.484066] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.492311] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.496025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.499017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.501022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.506000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.507723] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.510033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.513292] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.520021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.533025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.540020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.547798] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.557019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.565024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.588022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.605287] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.609018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.618024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.635031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.646181] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.652018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.658020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.674022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.687520] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.694018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.699037] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.715028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.722766] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.728018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.731024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.745023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.758937] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.768019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.784023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.804000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.820000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.823525] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.825403] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.826295] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.828154] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.832163] iommu: Default domain type: Passthrough [ 0.833468] SCSI subsystem initialized [ 0.837117] ACPI: bus type USB registered [ 0.838236] usbcore: registered new interface driver usbfs [ 0.841116] usbcore: registered new interface driver hub [ 0.842120] usbcore: registered new device driver usb [ 0.844417] pps_core: LinuxPPS API ver. 1 registered [ 0.845008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.848063] PTP clock support registered [ 0.849181] EDAC MC: Ver: 3.0.0 [ 0.851134] PCI: Using ACPI for IRQ routing [ 0.856731] NetLabel: Initializing [ 0.862017] NetLabel: domain hash size = 128 [ 0.863000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.864073] NetLabel: unlabeled traffic allowed by default [ 0.868455] vgaarb: loaded [ 0.871385] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.872017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.878799] clocksource: Switched to clocksource kvm-clock [ 1.088816] VFS: Disk quotas dquot_6.6.0 [ 1.090560] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.093424] *** VALIDATE ramfs *** [ 1.094881] *** VALIDATE hugetlbfs *** [ 1.098455] pnp: PnP ACPI init [ 1.105770] pnp: PnP ACPI: found 6 devices [ 1.132594] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.136667] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.139649] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.142565] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.145713] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.165265] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.170546] NET: Registered protocol family 2 [ 1.179424] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.185464] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.189929] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.197271] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.201804] TCP: Hash tables configured (established 65536 bind 65536) [ 1.205520] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.209281] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.222691] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.236962] NET: Registered protocol family 1 [ 1.240348] RPC: Registered named UNIX socket transport module. [ 1.243590] RPC: Registered udp transport module. [ 1.249601] RPC: Registered tcp transport module. [ 1.256932] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.260863] NET: Registered protocol family 44 [ 1.264353] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.267511] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.273507] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.284210] PCI: CLS 0 bytes, default 64 [ 1.287846] Unpacking initramfs... [ 4.549408] debug: unmapping init [mem 0xffff9f9b3cc54000-0xffff9f9b3ffbffff] [ 4.557794] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.568205] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.572503] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.612111] hrtimer: interrupt took 4098675 ns [ 6.798870] Initialise system trusted keyrings [ 6.804338] Key type blacklist registered [ 6.808222] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.837249] zbud: loaded [ 6.851581] *** VALIDATE nfs *** [ 6.856181] *** VALIDATE nfs4 *** [ 6.861298] pstore: using deflate compression [ 6.910289] Platform Keyring initialized [ 7.470538] NET: Registered protocol family 38 [ 7.473677] Key type asymmetric registered [ 7.487530] Asymmetric key parser 'x509' registered [ 7.504341] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.510794] io scheduler mq-deadline registered [ 7.514575] io scheduler kyber registered [ 7.517686] io scheduler bfq registered [ 7.522096] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.526135] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.531977] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.535444] ACPI: Power Button [PWRF] [ 7.548583] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.580311] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.635509] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.648089] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.678314] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.714174] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.741044] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.745647] Non-volatile memory driver v1.3 [ 7.747149] Linux agpgart interface v0.103 [ 7.802699] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 7.808098] vda: detected capacity change from 0 to 74702848 [ 7.854418] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.863249] vdb: detected capacity change from 0 to 1073741824 [ 7.914779] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.926775] vdc: detected capacity change from 0 to 2621440000 [ 7.975642] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.980078] vdd: detected capacity change from 0 to 2621440000 [ 8.010731] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.016164] vde: detected capacity change from 0 to 4294967296 [ 8.050741] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.068401] vdf: detected capacity change from 0 to 4294967296 [ 8.090393] libphy: Fixed MDIO Bus: probed [ 8.107357] usbcore: registered new interface driver usbserial_generic [ 8.112206] usbserial: USB Serial support registered for generic [ 8.115114] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.121337] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.123698] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.126318] mousedev: PS/2 mouse device common for all mice [ 8.130818] rtc_cmos 00:05: RTC can wake from S4 [ 8.135568] rtc_cmos 00:05: registered as rtc0 [ 8.139414] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.146938] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.150855] intel_pstate: CPU model not supported [ 8.168747] hid: raw HID events driver (C) Jiri Kosina [ 8.173508] usbcore: registered new interface driver usbhid [ 8.179317] usbhid: USB HID core driver [ 8.186234] drop_monitor: Initializing network drop monitor service [ 8.191691] Initializing XFRM netlink socket [ 8.193133] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.198823] NET: Registered protocol family 10 [ 8.228592] Segment Routing with IPv6 [ 8.231180] NET: Registered protocol family 17 [ 8.238291] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.240632] mpls_gso: MPLS GSO support [ 8.264864] RAS: Correctable Errors collector initialized. [ 8.268227] AVX version of gcm_enc/dec engaged. [ 8.270701] AES CTR mode by8 optimization enabled [ 8.384832] sched_clock: Marking stable (8384744807, 0)->(10652580477, -2267835670) [ 8.390134] registered taskstats version 1 [ 8.392700] Loading compiled-in X.509 certificates [ 8.395364] zswap: loaded using pool lzo/zbud [ 8.462156] Key type big_key registered [ 8.514553] Key type encrypted registered [ 8.521694] ima: No TPM chip found, activating TPM-bypass! [ 8.524410] ima: Allocated hash algorithm: sha1 [ 8.526562] ima: No architecture policies found [ 8.528676] evm: Initialising EVM extended attributes: [ 8.531227] evm: security.selinux [ 8.532557] evm: security.ima [ 8.534153] evm: security.capability [ 8.535849] evm: HMAC attrs: 0x1 [ 8.540384] rtc_cmos 00:05: setting system clock to 2026-08-14 15:15:20 UTC (1786720520) [ 8.562489] debug: unmapping init [mem 0xffffffffae203000-0xffffffffae3fffff] [ 8.575517] debug: unmapping init [mem 0xffffffffacf82000-0xffffffffad258fff] [ 8.594156] Write protecting the kernel read-only data: 28672k [ 8.601496] debug: unmapping init [mem 0xffffffffab603000-0xffffffffab7fffff] [ 8.615603] debug: unmapping init [mem 0xffffffffabf14000-0xffffffffabffffff] [ 8.766689] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.802271] systemd[1]: Detected virtualization kvm. [ 8.809670] systemd[1]: Detected architecture x86-64. [ 8.827651] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.927877] systemd[1]: No hostname configured. [ 8.929827] systemd[1]: Set hostname to . [ 8.946077] random: systemd: uninitialized urandom read (16 bytes read) [ 8.958225] systemd[1]: Initializing machine ID from random generator. [ 9.126271] random: ln: uninitialized urandom read (6 bytes read) [ 9.380436] random: systemd: uninitialized urandom read (16 bytes read) [ 9.399044] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 9.448079] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 9.476593] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.496287] device-mapper: uevent: version 1.0.3 [ 11.501799] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 14.073820] random: fast init done [ 14.133644] virtio_net virtio0 ens2: renamed from eth0 [ 14.288416] scsi host0: ata_piix [ 14.299182] scsi host1: ata_piix [ 14.303479] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.305892] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.051408] random: crng init done [ 19.061140] random: 7 urandom warning(s) missed due to ratelimiting [ 22.259030] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 23.867781] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.796049] printk: systemd: 24 output lines suppressed due to ratelimiting [ 27.753773] SELinux: Disabled at runtime. [ 27.857865] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.865765] systemd[1]: Detected virtualization kvm. [ 27.867383] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 29.513476] systemd[1]: initrd-switch-root.service: Succeeded. [ 29.517742] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 29.545799] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 29.552695] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 29.557448] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.570646] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.582454] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK [ 29.693189] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 30.683659] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.481932] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.646034] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.935607] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.976603] EDAC sbridge: Ver: 1.1.2 [ 35.413952] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit)[ 36.300954] NFS: Registering the id_resolver key type [ 36.304869] Key type id_resolver registered [ 36.306972] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg240-server login: [ 93.777205] libcfs: loading out-of-tree module taints kernel. [ 93.875758] Key type ._llcrypt registered [ 93.885309] Key type .llcrypt registered [ 94.077950] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 118.330561] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 120.028660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 120.060152] alg: No test for adler32 (adler32-zlib) [ 121.734969] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 122.912868] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 124.848660] Key type lgssc registered [ 126.925468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.247271] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 189.760861] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 203.864516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 203.887675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.231934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 205.265879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 205.348985] Lustre: lustre-MDT0000: new disk, initializing [ 205.530205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.604313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 210.562877] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.328281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.485567] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 225.518631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 225.523413] Lustre: Skipped 1 previous similar message [ 225.596953] Lustre: lustre-MDT0001: new disk, initializing [ 225.653559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.684569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 225.695353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 230.508554] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.917579] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 248.998017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.506440] Lustre: lustre-OST0000: new disk, initializing [ 249.529941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 249.555072] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.651568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.549097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 255.575403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 255.709778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 256.989454] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.603200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.731274] Lustre: lustre-OST0001: new disk, initializing [ 271.736747] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 271.741893] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.795357] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 277.758748] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.660317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 281.677132] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 281.773505] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 290.822946] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 297.053510] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 303.296908] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing check_logdir /tmp/testlogs/ [ 308.775245] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing yml_node [ 314.187312] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 317.423543] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 321.079874] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 323.122682] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Fri Aug 14 11:20:33 EDT 2026 [ 345.441926] Lustre: DEBUG MARKER: excepting tests: 18b 23b [ 355.987834] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 364.002984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 364.007273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.024339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 368.610242] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.610603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 368.618296] Lustre: Skipped 2 previous similar messages [ 368.631185] Lustre: Skipped 3 previous similar messages [ 370.254925] Lustre: server umount lustre-MDT0000 complete [ 378.852415] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 378.862932] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 379.708088] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786720891 with bad export cookie 10816305676572804243 [ 379.720514] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 379.965389] Lustre: server umount lustre-MDT0001 complete [ 399.201167] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786720895/real 1786720895] req@ffff9f9bb5e1e680 x1873512375621888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786720911 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 399.227904] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 400.353355] Lustre: 3645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786720896/real 1786720896] req@ffff9f9bb5e1d180 x1873512375622144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786720912 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 400.419538] Lustre: server umount lustre-OST0000 complete [ 404.389303] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786720900/real 1786720900] req@ffff9f9bb5e1f100 x1873512375622400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786720916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 409.609645] Lustre: server umount lustre-OST0001 complete [ 430.126744] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 433.035514] Key type lgssc unregistered [ 433.395666] LNet: 14800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 433.417603] LNetError: 14800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 433.458242] LNet: Removed LNI 192.168.202.140@tcp [ 434.606180] Key type .llcrypt unregistered [ 434.611085] Key type ._llcrypt unregistered [ 460.093945] Key type ._llcrypt registered [ 460.095277] Key type .llcrypt registered [ 460.174765] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 475.266962] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 476.485699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 476.574717] alg: No test for adler32 (adler32-zlib) [ 477.731278] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 478.019683] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 479.712214] Key type lgssc registered [ 481.775958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 538.654589] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 553.674989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 553.728717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.151719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 555.198529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 555.299818] Lustre: lustre-MDT0000: new disk, initializing [ 555.380479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.396374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 561.751208] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 577.611493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.756056] Lustre: 19254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 577.800024] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 577.807848] Lustre: Skipped 1 previous similar message [ 577.927123] Lustre: lustre-MDT0001: new disk, initializing [ 577.989865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 578.022408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 578.033591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 583.259535] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 588.579519] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 599.200939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 599.538969] Lustre: lustre-OST0000: new disk, initializing [ 599.544972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 599.552225] Lustre: 21196:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 599.593614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 599.740787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 599.758225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 599.883891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 607.329908] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 622.264586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 622.465145] Lustre: lustre-OST0001: new disk, initializing [ 622.473556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 622.487865] Lustre: 22221:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 622.593864] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 629.433396] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 630.890547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 630.904320] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 630.958179] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 641.853462] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 649.695849] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 657.596919] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 11:26:08 (1786721168) === [ 660.539460] Lustre: DEBUG MARKER: == sanity-lfsck test 0: Control LFSCK manually =========== 11:26:11 (1786721171) [ 660.730960] Lustre: 21660:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 660.747594] Lustre: 21660:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 660.761420] Lustre: 21660:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 660.782029] Lustre: 21660:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 660.787497] Lustre: 21660:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 660.812129] Lustre: 21660:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 661.386809] Lustre: 21660:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 661.393677] Lustre: 21660:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 11 previous similar messages [ 661.399398] Lustre: 21660:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 661.404734] Lustre: 21660:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 661.410595] Lustre: 21660:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 661.420117] Lustre: 21660:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 661.427099] Lustre: 21660:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 661.436928] Lustre: 21660:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 661.449790] Lustre: 21660:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 661.468117] Lustre: 21660:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 661.474406] Lustre: 21660:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 661.482295] Lustre: 21660:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 662.400804] Lustre: 19263:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 662.409874] Lustre: 19263:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 38 previous similar messages [ 662.416176] Lustre: 19263:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 662.423300] Lustre: 19263:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 662.429044] Lustre: 19263:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 662.439896] Lustre: 19263:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 662.446767] Lustre: 19263:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 662.453523] Lustre: 19263:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 662.459793] Lustre: 19263:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 662.466080] Lustre: 19263:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 662.529355] Lustre: 19264:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 662.536981] Lustre: 19264:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 41 previous similar messages [ 664.407196] Lustre: 19262:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 664.418354] Lustre: 19262:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 95 previous similar messages [ 664.425128] Lustre: 19262:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 664.432237] Lustre: 19262:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 664.440443] Lustre: 19262:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 664.451167] Lustre: 19262:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 664.458355] Lustre: 19262:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 664.466050] Lustre: 19262:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 664.472447] Lustre: 19262:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 664.478197] Lustre: 19262:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 95 previous similar messages [ 664.578922] Lustre: 19262:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 664.586311] Lustre: 19262:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 98 previous similar messages [ 668.563438] Lustre: *** cfs_fail_loc=1600, val=3*** [ 670.937633] Lustre: 21185:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 274, rollback = 2 [ 670.948205] Lustre: 23411:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 670.951882] Lustre: 21185:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 131 previous similar messages [ 670.951917] Lustre: 21185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 670.951921] Lustre: 21185:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 130 previous similar messages [ 670.951927] Lustre: 21185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 670.951931] Lustre: 21185:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 130 previous similar messages [ 670.951962] Lustre: 21185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 670.951966] Lustre: 21185:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 130 previous similar messages [ 670.951971] Lustre: 21185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 670.951974] Lustre: 21185:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 124 previous similar messages [ 671.089397] Lustre: 23411:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 131 previous similar messages [ 673.212030] Lustre: *** cfs_fail_loc=1600, val=3*** [ 683.687027] Lustre: 23619:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0001: opcode 2: before 259 < left 274, rollback = 2 [ 683.713073] Lustre: 23619:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 59 previous similar messages [ 683.721906] Lustre: 23619:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 683.730697] Lustre: 23622:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 683.733030] Lustre: 23619:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 60 previous similar messages [ 683.733059] Lustre: 23619:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 683.733063] Lustre: 23619:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 60 previous similar messages [ 683.733069] Lustre: 23619:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 683.733072] Lustre: 23619:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 60 previous similar messages [ 683.733078] Lustre: 23619:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 683.733081] Lustre: 23619:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 60 previous similar messages [ 683.837582] Lustre: 23622:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 64 previous similar messages [ 687.586498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.591040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 687.605066] Lustre: Skipped 1 previous similar message [ 687.641246] Lustre: Skipped 3 previous similar messages [ 692.709211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 692.714632] Lustre: Skipped 3 previous similar messages [ 697.826938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 697.838699] Lustre: Skipped 3 previous similar messages [ 700.384173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 700.660629] Lustre: server umount lustre-MDT0000 complete [ 704.177962] LustreError: 19247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786721216 with bad export cookie 11915009424671385997 [ 704.184182] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.192647] LustreError: 19247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 704.552913] Lustre: server umount lustre-MDT0001 complete [ 718.799919] Lustre: server umount lustre-OST0000 complete [ 734.150349] Lustre: server umount lustre-OST0001 complete [ 743.957693] Lustre: DEBUG MARKER: == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 11:27:34 (1786721254) [ 760.360379] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 770.030920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.396959] LustreError: 26230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 770.420990] LustreError: 26230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 770.520943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.313281] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 775.661424] LustreError: 26231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 780.772540] LustreError: 26230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 783.697406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.034861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 788.709472] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 792.018123] Lustre: 27370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 798.568401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 805.009945] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 808.102749] LustreError: 27724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.474568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 813.545968] LustreError: 27725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.685756] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 813.695507] Lustre: Skipped 1 previous similar message [ 815.741628] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:65) [ 815.759339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:40 to 0x280000401:65) [ 820.025993] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 827.981703] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 831.498258] Lustre: 29239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 833.014397] Lustre: 26227:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 833.020575] Lustre: 26227:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 7 previous similar messages [ 833.029544] Lustre: 26227:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 833.037401] Lustre: 26227:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 833.047179] Lustre: 26227:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 833.051121] Lustre: 26227:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 833.061582] Lustre: 26227:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 833.065619] Lustre: 26227:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 833.071699] Lustre: 26227:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 833.078469] Lustre: 26227:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 833.087418] Lustre: 26227:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 833.097476] Lustre: 26227:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 838.265491] Lustre: *** cfs_fail_loc=1501, val=0*** [ 845.074537] Lustre: Failing over lustre-MDT0000 [ 845.284234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 845.289451] Lustre: server umount lustre-MDT0000 complete [ 845.298125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 845.298133] Lustre: Skipped 2 previous similar messages [ 845.299858] LustreError: 29253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.821977] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 846.832288] Lustre: Skipped 2 previous similar messages [ 855.003282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 855.177323] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 855.391953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 855.436551] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 860.145328] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 860.643735] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 860.647534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 860.679493] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 860.733168] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 860.734713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 863.382927] Lustre: *** cfs_fail_loc=1505, val=0*** [ 870.984557] Lustre: DEBUG MARKER: == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 11:29:41 (1786721381) [ 872.235131] Lustre: 29249:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 872.242833] Lustre: 29249:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 321 previous similar messages [ 872.248330] Lustre: 29249:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 872.252888] Lustre: 29249:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 872.258311] Lustre: 29249:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 872.264320] Lustre: 29249:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 872.269804] Lustre: 29249:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 872.277270] Lustre: 29249:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 872.283139] Lustre: 29249:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 872.287300] Lustre: 29249:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 872.293812] Lustre: 29249:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 872.302028] Lustre: 29249:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 877.478294] Lustre: *** cfs_fail_loc=1502, val=0*** [ 887.346325] Lustre: Failing over lustre-MDT0000 [ 887.524817] Lustre: server umount lustre-MDT0000 complete [ 891.360946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 891.374047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 891.383822] LustreError: 26227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 891.393880] Lustre: Skipped 3 previous similar messages [ 891.406372] LustreError: 26227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 897.541260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 897.666726] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 897.996841] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 902.340665] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 903.146120] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 903.146584] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 903.161876] Lustre: Skipped 3 previous similar messages [ 903.218233] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 903.280471] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:169 to 0x2c0000401:193) [ 903.280537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 905.896313] Lustre: *** cfs_fail_loc=1505, val=0*** [ 912.868928] Lustre: DEBUG MARKER: == sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 11:30:23 (1786721423) [ 918.331674] Lustre: *** cfs_fail_loc=1504, val=0*** [ 918.336059] Lustre: *** cfs_fail_loc=1504, val=0*** [ 918.338105] Lustre: Skipped 1 previous similar message [ 926.148164] Lustre: Failing over lustre-MDT0000 [ 926.455683] Lustre: server umount lustre-MDT0000 complete [ 928.737723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 928.738745] LustreError: 26230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 928.761072] Lustre: Skipped 2 previous similar messages [ 928.778415] LustreError: 26230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 935.789339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 935.906628] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 936.174749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 936.177300] Lustre: Skipped 1 previous similar message [ 936.219276] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 940.815947] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 941.538703] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 941.545264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 941.548618] Lustre: Skipped 3 previous similar messages [ 941.578308] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 941.647777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:233 to 0x280000401:257) [ 941.651148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:257) [ 944.501594] Lustre: *** cfs_fail_loc=1505, val=0*** [ 952.638965] Lustre: DEBUG MARKER: == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 11:31:03 (1786721463) [ 953.781334] Lustre: 26225:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 953.789249] Lustre: 26225:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 644 previous similar messages [ 953.794584] Lustre: 26225:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 953.800194] Lustre: 26225:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 953.803319] Lustre: 26225:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 953.808475] Lustre: 26225:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 953.812564] Lustre: 26225:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 953.816528] Lustre: 26225:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 953.821276] Lustre: 26225:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 953.828462] Lustre: 26225:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 953.836502] Lustre: 26225:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 953.842950] Lustre: 26225:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 959.499785] Lustre: *** cfs_fail_loc=1603, val=0*** [ 967.947232] Lustre: Failing over lustre-MDT0000 [ 968.175281] Lustre: server umount lustre-MDT0000 complete [ 972.256793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 972.270822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.293569] Lustre: Skipped 4 previous similar messages [ 978.576963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 978.705614] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 978.918905] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 983.068284] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 984.039543] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 984.059549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 984.069727] Lustre: Skipped 3 previous similar messages [ 984.092613] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 984.151348] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:297 to 0x2c0000401:321) [ 984.154450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:297 to 0x280000401:321) [ 992.016175] Lustre: DEBUG MARKER: == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 11:31:42 (1786721502) [ 997.743622] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1004.850694] Lustre: Failing over lustre-MDT0000 [ 1005.173731] Lustre: server umount lustre-MDT0000 complete [ 1009.633060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1009.635793] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.652134] Lustre: Skipped 1 previous similar message [ 1009.657125] LustreError: 26225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1009.680762] LustreError: 26225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 1015.422284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1015.533343] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1015.862617] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1019.931246] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1020.899037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1020.901582] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1020.915844] Lustre: Skipped 3 previous similar messages [ 1020.981521] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1021.026677] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:361 to 0x2c0000401:385) [ 1021.026818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:361 to 0x280000401:385) [ 1029.703693] Lustre: DEBUG MARKER: == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 11:32:20 (1786721540) [ 1035.186315] Lustre: *** cfs_fail_loc=1605, val=0*** [ 1044.025475] Lustre: Failing over lustre-MDT0000 [ 1044.424384] Lustre: server umount lustre-MDT0000 complete [ 1046.505927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1046.513964] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.544894] Lustre: Skipped 5 previous similar messages [ 1059.278330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1059.533815] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1060.034290] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1065.468122] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1065.488674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1065.502581] Lustre: Skipped 3 previous similar messages [ 1065.536922] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1065.614849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:425 to 0x280000401:449) [ 1065.615119] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:425 to 0x2c0000401:449) [ 1068.080331] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1080.810279] Lustre: DEBUG MARKER: == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 11:33:10 (1786721590) [ 1083.137894] Lustre: 26226:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1083.154032] Lustre: 26226:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 968 previous similar messages [ 1083.164122] Lustre: 26226:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1083.181524] Lustre: 26226:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1083.195164] Lustre: 26226:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1083.217679] Lustre: 26226:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1083.236305] Lustre: 26226:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1083.251902] Lustre: 26226:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1083.264302] Lustre: 26226:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1083.273234] Lustre: 26226:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1083.283907] Lustre: 26226:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1083.296771] Lustre: 26226:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1089.027768] Lustre: *** cfs_fail_loc=161d, val=0*** [ 1096.866038] Lustre: Failing over lustre-MDT0000 [ 1097.216949] Lustre: server umount lustre-MDT0000 complete [ 1101.287617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1108.094107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1108.348916] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.726563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.734432] Lustre: Skipped 3 previous similar messages [ 1108.804349] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1114.082704] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1114.085686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1114.103266] Lustre: Skipped 3 previous similar messages [ 1114.122913] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1114.175874] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 1114.175874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:489 to 0x280000401:513) [ 1114.308860] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1124.055167] Lustre: DEBUG MARKER: == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 11:33:54 (1786721634) [ 1126.899992] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1137.044778] Lustre: DEBUG MARKER: == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 11:34:07 (1786721647) [ 1142.371848] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1143.477664] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1153.613884] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 11:34:24 (1786721664) [ 1188.527341] Lustre: Failing over lustre-MDT0000 [ 1188.893139] Lustre: server umount lustre-MDT0000 complete [ 1190.881633] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.884340] LustreError: 26227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.888468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1190.888550] Lustre: Skipped 7 previous similar messages [ 1190.935019] LustreError: 26227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 1193.484513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1204.323034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1206.737106] Lustre: 16415:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786721702/real 1786721702] req@ffff9f9b812ac380 x1873512748489344/t0(0) o400->MGC192.168.202.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786721718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1206.792317] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1215.353142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1215.369148] Lustre: lustre-MDT0000: reset Object Index mappings [ 1216.480693] LustreError: 42397:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1216.490065] LustreError: 42397:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9f9babfed500 x1873512748495360/t0(0) o250->MGC192.168.202.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786721728 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1216.535787] LustreError: 42397:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1217.227270] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1221.237903] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1222.628965] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1222.630158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1222.644630] Lustre: Skipped 3 previous similar messages [ 1222.668467] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1222.707861] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:609) [ 1222.708109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:609) [ 1224.795742] LustreError: 42883:0:(lfsck_engine.c:1018:lfsck_master_engine()) lustre-MDT0000-osd: master engine fail to verify the .lustre/lost+found/, go ahead: rc = -115 [ 1224.819838] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1226.848350] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1226.856689] Lustre: Skipped 1 previous similar message [ 1235.297859] Lustre: Failing over lustre-MDT0000 [ 1235.430596] Lustre: server umount lustre-MDT0000 complete [ 1238.001524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1245.954213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1250.046790] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1251.342894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:641) [ 1251.347481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:641) [ 1252.616980] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1259.156646] Lustre: DEBUG MARKER: == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 11:36:10 (1786721770) [ 1261.469822] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1269.926070] Lustre: Failing over lustre-MDT0000 [ 1272.075445] Lustre: server umount lustre-MDT0000 complete [ 1276.501638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1286.521524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1287.136976] Lustre: 16416:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786721783/real 1786721783] req@ffff9f9babe21f80 x1873512748580480/t0(0) o400->MGC192.168.202.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786721799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1297.506526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1297.527680] Lustre: lustre-MDT0000: reset Object Index mappings [ 1312.954230] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1312.967696] Lustre: Skipped 1 previous similar message [ 1317.521144] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1318.377568] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1318.380505] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1318.387726] Lustre: Skipped 1 previous similar message [ 1318.405809] Lustre: Skipped 7 previous similar messages [ 1318.431551] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1318.443961] Lustre: Skipped 1 previous similar message [ 1318.485396] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:705) [ 1318.492909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:705) [ 1321.109750] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1321.112071] Lustre: Skipped 2 previous similar messages [ 1333.023042] Lustre: Failing over lustre-MDT0000 [ 1333.296835] Lustre: server umount lustre-MDT0000 complete [ 1333.732246] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.743989] Lustre: Skipped 10 previous similar messages [ 1333.745135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1343.258601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1343.451544] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1343.461953] LustreError: Skipped 2 previous similar messages [ 1349.098151] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1349.170478] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:737) [ 1349.174431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:737) [ 1353.431467] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1353.435032] Lustre: Skipped 84 previous similar messages [ 1363.071173] Lustre: DEBUG MARKER: == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 11:37:53 (1786721873) [ 1364.565245] Lustre: 40986:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 278, rollback = 2 [ 1364.574959] Lustre: 40986:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1282 previous similar messages [ 1364.582275] Lustre: 40986:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 1364.588530] Lustre: 40986:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1283 previous similar messages [ 1364.595521] Lustre: 40986:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1364.605629] Lustre: 40986:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1283 previous similar messages [ 1364.615285] Lustre: 40986:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1364.622734] Lustre: 40986:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1283 previous similar messages [ 1364.630764] Lustre: 40986:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 1364.641729] Lustre: 40986:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1283 previous similar messages [ 1364.659449] Lustre: 40986:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1364.666745] Lustre: 40986:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1283 previous similar messages [ 1371.594889] Lustre: *** cfs_fail_loc=1600, val=1*** [ 1371.596833] Lustre: Skipped 7 previous similar messages [ 1390.495482] Lustre: DEBUG MARKER: == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 11:38:21 (1786721901) [ 1397.745611] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1397.749106] Lustre: Skipped 8 previous similar messages [ 1425.600708] Lustre: DEBUG MARKER: == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 11:38:55 (1786721935) [ 1439.920970] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1439.923403] Lustre: Skipped 13 previous similar messages [ 1445.786894] Lustre: Failing over lustre-MDT0000 [ 1446.033565] Lustre: server umount lustre-MDT0000 complete [ 1451.489945] LustreError: 30849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.522731] LustreError: 30849:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 79 previous similar messages [ 1456.431755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1456.894765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.902258] Lustre: Skipped 4 previous similar messages [ 1456.964783] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1456.982986] Lustre: Skipped 1 previous similar message [ 1461.595394] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1462.247850] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1462.265513] Lustre: Skipped 1 previous similar message [ 1462.277852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1462.285394] Lustre: Skipped 7 previous similar messages [ 1462.314349] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1462.321605] Lustre: Skipped 1 previous similar message [ 1462.379543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:853 to 0x280000401:897) [ 1462.386523] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:854 to 0x2c0000401:897) [ 1472.754513] Lustre: DEBUG MARKER: == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 11:39:43 (1786721983) [ 1488.234371] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1507.166722] Lustre: 52971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1526.127459] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1529.057823] Lustre: 54106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1534.594201] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1534.597964] Lustre: Skipped 81 previous similar messages [ 1537.045639] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1538.085653] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1539.104198] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1541.152226] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1541.154514] Lustre: Skipped 1 previous similar message [ 1542.219788] Lustre: Failing over lustre-MDT0000 [ 1542.477425] Lustre: server umount lustre-MDT0000 complete [ 1544.160792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1544.174186] LustreError: Skipped 1 previous similar message [ 1551.956680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1556.426588] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1557.582791] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:946 to 0x2c0000401:961) [ 1557.591273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:947 to 0x280000401:993) [ 1566.207652] Lustre: DEBUG MARKER: == sanity-lfsck test 8: LFSCK state machine ============== 11:41:17 (1786722077) [ 1572.844249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1572.854586] Lustre: Skipped 2 previous similar messages [ 1574.786960] Lustre: server umount lustre-MDT0000 complete [ 1578.244175] LustreError: 26213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786722090 with bad export cookie 11915009424671599091 [ 1578.261492] LustreError: 26213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1578.636749] Lustre: server umount lustre-MDT0001 complete [ 1592.408272] Lustre: server umount lustre-OST0000 complete [ 1606.224735] Lustre: server umount lustre-OST0001 complete [ 1612.613515] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 1620.279182] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1665.838679] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1675.973612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1676.230154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1676.259369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1676.315669] Lustre: lustre-MDT0000: new disk, initializing [ 1676.375834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1680.763938] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1690.369236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.467071] Lustre: 59166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1690.502569] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1690.509097] Lustre: Skipped 1 previous similar message [ 1690.558645] Lustre: lustre-MDT0001: new disk, initializing [ 1690.652894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1690.663763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1695.892565] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1701.055350] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1707.604864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1707.827230] Lustre: lustre-OST0000: new disk, initializing [ 1707.830937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1707.838510] Lustre: 60799:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1709.257401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1709.268107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1709.319632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1713.353199] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1723.931749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1724.094148] Lustre: lustre-OST0001: new disk, initializing [ 1724.097896] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1724.102656] Lustre: 61669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1725.566710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1725.590860] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1725.763144] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1731.367132] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1745.651783] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1750.636309] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1760.523184] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1761.403660] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1761.415420] Lustre: Skipped 19 previous similar messages [ 1765.507646] Lustre: *** cfs_fail_loc=1601, val=2*** [ 1765.515404] Lustre: Skipped 5 previous similar messages [ 1782.450406] Lustre: Failing over lustre-MDT0000 [ 1782.762052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.791754] Lustre: Skipped 13 previous similar messages [ 1782.951964] Lustre: server umount lustre-MDT0000 complete [ 1792.471694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1792.551722] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1792.556218] LustreError: Skipped 3 previous similar messages [ 1792.717518] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1792.727632] Lustre: Skipped 1 previous similar message [ 1797.098855] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1798.115876] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1798.121320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1798.130145] Lustre: Skipped 1 previous similar message [ 1798.142269] Lustre: Skipped 7 previous similar messages [ 1798.158774] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1798.165293] Lustre: Skipped 1 previous similar message [ 1798.199520] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1798.205846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:65) [ 1798.206761] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:65) [ 1805.781397] Lustre: Failing over lustre-MDT0000 [ 1808.302365] Lustre: server umount lustre-MDT0000 complete [ 1808.368384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1808.378100] LustreError: Skipped 2 previous similar messages [ 1819.833049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1825.314393] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1825.328681] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:97) [ 1825.329792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:97) [ 1826.899615] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1832.553835] Lustre: Failing over lustre-MDT0000 [ 1832.971868] Lustre: server umount lustre-MDT0000 complete [ 1842.397677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1848.389990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:129) [ 1848.395291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:129) [ 1848.634117] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1854.288629] Lustre: *** cfs_fail_loc=1602, val=2*** [ 1854.290731] Lustre: Skipped 1 previous similar message [ 1867.176181] Lustre: DEBUG MARKER: == sanity-lfsck test 9a: LFSCK speed control (1) ========= 11:46:18 (1786722378) [ 1880.854054] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1900.284643] Lustre: 68624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1927.002612] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1931.199488] Lustre: 69761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1939.284653] Lustre: 59175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1939.290771] Lustre: 59175:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 2828 previous similar messages [ 1939.295235] Lustre: 59175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1939.301570] Lustre: 59175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 2829 previous similar messages [ 1939.312122] Lustre: 59175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1939.320870] Lustre: 59175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 2829 previous similar messages [ 1939.328292] Lustre: 59175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1939.334673] Lustre: 59175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 2829 previous similar messages [ 1939.341976] Lustre: 59175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1939.348143] Lustre: 59175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 2828 previous similar messages [ 1939.356267] Lustre: 59175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1939.361102] Lustre: 59175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 2828 previous similar messages [ 2045.778904] Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) ========= 11:49:16 (1786722556) [ 2087.887673] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2087.893757] Lustre: Skipped 4 previous similar messages [ 2116.207155] Lustre: *** cfs_fail_loc=160c, val=0*** [ 2116.209959] Lustre: Skipped 7 previous similar messages [ 2151.751358] Lustre: DEBUG MARKER: == sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 11:51:02 (1786722662) [ 2190.478376] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2191.482365] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2191.484855] Lustre: Skipped 49 previous similar messages [ 2193.486429] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2193.489556] Lustre: Skipped 97 previous similar messages [ 2197.487324] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2197.489701] Lustre: Skipped 266 previous similar messages [ 2205.536709] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2205.543099] Lustre: Skipped 356 previous similar messages [ 2221.555466] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2221.558221] Lustre: Skipped 852 previous similar messages [ 2253.562277] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2253.567986] Lustre: Skipped 1756 previous similar messages [ 2256.754730] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2256.758471] Lustre: Skipped 2599 previous similar messages [ 2489.607734] Lustre: DEBUG MARKER: == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 11:56:40 (1786723000) [ 2628.666620] Lustre: 61677:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 2628.683449] Lustre: 61677:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 36832 previous similar messages [ 2628.691858] Lustre: 61677:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 2628.698712] Lustre: 61677:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 36832 previous similar messages [ 2628.704501] Lustre: 61677:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 2628.712428] Lustre: 61677:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 36832 previous similar messages [ 2628.719863] Lustre: 61677:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 2628.727457] Lustre: 61677:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 36832 previous similar messages [ 2628.734922] Lustre: 61677:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 2628.740908] Lustre: 61677:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 36832 previous similar messages [ 2628.747292] Lustre: 61677:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 2628.754426] Lustre: 61677:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 36832 previous similar messages [ 2636.770138] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.773143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2636.779794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2636.793218] Lustre: Skipped 13 previous similar messages [ 2641.649169] Lustre: server umount lustre-MDT0000 complete [ 2641.892552] LustreError: 70189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.904962] LustreError: 70189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 48 previous similar messages [ 2645.800802] LustreError: 59158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786723157 with bad export cookie 11915009424671618187 [ 2645.803023] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2645.815135] LustreError: 59158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2645.829085] LustreError: Skipped 2 previous similar messages [ 2646.199950] Lustre: server umount lustre-MDT0001 complete [ 2662.015562] Lustre: server umount lustre-OST0000 complete [ 2662.883886] Lustre: 16414:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786723158/real 1786723158] req@ffff9f9b873ddc00 x1873512752673920/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786723174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2665.441414] Lustre: 16416:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786723161/real 1786723161] req@ffff9f9b873dd880 x1873512752674176/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786723177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2666.684490] Lustre: server umount lustre-OST0001 complete [ 2674.317969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 2685.616940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2701.281207] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2706.400693] LustreError: 74959:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.140@tcp: failed processing log, type 4: rc = -110 [ 2732.131780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.153211] Lustre: Skipped 8 previous similar messages [ 2741.589237] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2747.739147] Lustre: 75544:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2747.765370] Lustre: *** cfs_fail_loc=160e, val=3*** [ 2750.850202] Lustre: 75544:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2763.189308] Lustre: DEBUG MARKER: == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 12:01:13 (1786723273) [ 2779.663843] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 2791.753062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.087672] LustreError: 74984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.219095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3271 to 0x280000401:3297) [ 2796.976360] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2806.475990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.019312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2807.029732] Lustre: Skipped 1 previous similar message [ 2813.326737] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2817.090979] Lustre: 78173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2838.850920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2844.665545] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3208 to 0x2c0000401:3233) [ 2846.926226] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2858.023654] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2864.504375] Lustre: 79675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2870.962492] Lustre: *** cfs_fail_loc=160d, val=0*** [ 2871.892488] Lustre: *** cfs_fail_loc=160d, val=0*** [ 2871.894122] Lustre: Skipped 3 previous similar messages [ 2879.847856] Lustre: Failing over lustre-OST0000 [ 2879.977505] Lustre: server umount lustre-OST0000 complete [ 2892.342193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2892.537716] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2892.544714] Lustre: Skipped 2 previous similar messages [ 2894.500193] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2894.518866] Lustre: Skipped 2 previous similar messages [ 2894.543667] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2894.547252] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2894.548168] Lustre: *** cfs_fail_loc=215, val=0*** [ 2894.559634] Lustre: Skipped 2 previous similar messages [ 2894.587640] Lustre: Skipped 11 previous similar messages [ 2899.936679] Lustre: *** cfs_fail_loc=215, val=0*** [ 2899.948070] Lustre: Skipped 2 previous similar messages [ 2901.008925] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2905.056543] Lustre: *** cfs_fail_loc=215, val=0*** [ 2905.059459] Lustre: Skipped 1 previous similar message [ 2905.818202] Lustre: 81077:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2905.881078] Lustre: 81077:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2909.508342] Lustre: Failing over lustre-OST0000 [ 2909.835847] Lustre: server umount lustre-OST0000 complete [ 2921.111699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2922.606223] Lustre: *** cfs_fail_loc=215, val=0*** [ 2922.606223] Lustre: *** cfs_fail_loc=215, val=0*** [ 2922.615298] Lustre: Skipped 2 previous similar messages [ 2928.096720] Lustre: *** cfs_fail_loc=215, val=0*** [ 2929.694978] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2939.891238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2939.902210] Lustre: Skipped 3 previous similar messages [ 2941.953855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2941.959876] Lustre: Skipped 3 previous similar messages [ 2944.135843] Lustre: server umount lustre-MDT0000 complete [ 2947.043850] LustreError: 77980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.072851] LustreError: 77980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 34 previous similar messages [ 2948.497657] LustreError: 74966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786723460 with bad export cookie 11915009424673190541 [ 2948.507241] LustreError: 74966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2948.793583] Lustre: server umount lustre-MDT0001 complete [ 2965.286749] Lustre: server umount lustre-OST0000 complete [ 2968.544475] Lustre: 16417:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786723464/real 1786723464] req@ffff9f9bbb83b100 x1873512752783104/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786723480 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2970.700927] Lustre: server umount lustre-OST0001 complete [ 2983.418900] Lustre: DEBUG MARKER: == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 12:04:53 (1786723493) [ 3004.730619] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3017.305776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.085769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3018.102865] Lustre: Skipped 3 previous similar messages [ 3023.443523] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3033.754550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.439573] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3042.967801] Lustre: 85464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3051.064868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3059.701972] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3062.614786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3362 to 0x280000401:3393) [ 3070.384136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3076.094814] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3208 to 0x2c0000401:3265) [ 3078.329841] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3088.247090] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3093.791936] Lustre: 87334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3173.917395] Lustre: DEBUG MARKER: == sanity-lfsck test 12b: auto detect Lustre device ====== 12:08:03 (1786723683) [ 3194.772227] Lustre: DEBUG MARKER: == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 12:08:24 (1786723704) [ 3196.217303] Lustre: *** cfs_fail_loc=160f, val=0*** [ 3208.410307] Lustre: DEBUG MARKER: == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 12:08:38 (1786723718) [ 3213.728252] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3213.732909] Lustre: Skipped 3 previous similar messages [ 3232.108608] Lustre: 88115:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 252 < left 260, rollback = 2 [ 3232.138827] Lustre: 88115:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1323 previous similar messages [ 3232.160330] Lustre: 88115:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3232.190056] Lustre: 88115:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1323 previous similar messages [ 3232.231325] Lustre: 88115:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/260/0 [ 3232.279835] Lustre: 88115:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1323 previous similar messages [ 3232.307419] Lustre: 88115:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 0/0/0, quota 1/3/2 [ 3232.331426] Lustre: 88115:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1323 previous similar messages [ 3232.352454] Lustre: 88115:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 1/17/2, delete: 0/0/0 [ 3232.363574] Lustre: 88115:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1323 previous similar messages [ 3232.373101] Lustre: 88115:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3232.395756] Lustre: 88115:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1323 previous similar messages [ 3270.626985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.642292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3270.650655] Lustre: Skipped 13 previous similar messages [ 3275.754912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3275.763475] Lustre: Skipped 6 previous similar messages [ 3277.479382] Lustre: server umount lustre-MDT0000 complete [ 3280.889110] LustreError: 87347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3280.918643] LustreError: 87347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 3281.428107] LustreError: 84305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786723793 with bad export cookie 11915009424673199004 [ 3281.438884] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3281.462476] LustreError: 84305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3281.491240] LustreError: Skipped 1 previous similar message [ 3281.847223] Lustre: server umount lustre-MDT0001 complete [ 3295.331538] Lustre: server umount lustre-OST0000 complete [ 3309.608362] Lustre: server umount lustre-OST0001 complete [ 3325.645786] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3335.236324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.682078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.685505] Lustre: Skipped 3 previous similar messages [ 3340.171157] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3349.980953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.713972] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3357.928530] Lustre: 93964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3364.837051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.736871] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3375.529845] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:161) [ 3380.648293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3554 to 0x280000401:3585) [ 3381.161104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3386.858636] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3361) [ 3386.862790] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:161) [ 3387.691440] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3395.154479] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3398.557608] Lustre: 95834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3405.802578] Lustre: DEBUG MARKER: == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 12:11:56 (1786723916) [ 3411.551027] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3411.555226] Lustre: Skipped 63 previous similar messages [ 3442.657066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3442.673151] LustreError: Skipped 3 previous similar messages [ 3442.681609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3445.314395] Lustre: server umount lustre-MDT0000 complete [ 3448.707671] LustreError: 92805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786723960 with bad export cookie 11915009424673227424 [ 3448.737693] LustreError: 92805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3448.972428] Lustre: server umount lustre-MDT0001 complete [ 3462.804728] Lustre: server umount lustre-OST0000 complete [ 3477.544457] Lustre: server umount lustre-OST0001 complete [ 3502.181728] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3515.187403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.993464] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3531.507290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.850406] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3542.059527] Lustre: 99880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3550.132373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3557.619707] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:193) [ 3558.326180] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3560.724619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3682 to 0x280000401:3713) [ 3569.060597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3574.765692] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3393) [ 3574.770061] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:193) [ 3577.733812] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3587.240398] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3602.024280] Lustre: DEBUG MARKER: == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 12:15:11 (1786724111) [ 3606.244348] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3606.245450] Lustre: Skipped 63 previous similar messages [ 3606.526688] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3626.413198] Lustre: DEBUG MARKER: == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 12:15:35 (1786724135) [ 3629.339933] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3629.467700] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3629.469163] Lustre: Skipped 2 previous similar messages [ 3644.056615] Lustre: DEBUG MARKER: == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 12:15:54 (1786724154) [ 3646.537914] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_15c MDS newer than 2.7.55, LU-6475 [ 3648.853649] Lustre: DEBUG MARKER: == sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 12:15:59 (1786724159) [ 3655.461800] Lustre: *** cfs_fail_loc=1709, val=0*** [ 3655.476261] LustreError: 98746:0:(mdt_reint.c:2644:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240002340:0x6c:0x0]/f50 failed: rc = -5 [ 3659.768069] Lustre: *** cfs_fail_loc=1709, val=0*** [ 3742.177870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3742.187056] Lustre: Skipped 4 previous similar messages [ 3746.498495] Lustre: server umount lustre-MDT0000 complete [ 3757.776407] LustreError: 98719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786724269 with bad export cookie 11915009424673242159 [ 3757.794743] LustreError: 98719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3758.214641] Lustre: server umount lustre-MDT0001 complete [ 3775.269964] Lustre: 16414:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724271/real 1786724271] req@ffff9f9a881f1500 x1873512753837056/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786724287 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3775.309273] Lustre: 16414:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3778.528236] Lustre: 16416:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724274/real 1786724274] req@ffff9f9bbbb3d500 x1873512753837312/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786724290 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3780.449085] Lustre: 16414:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724276/real 1786724276] req@ffff9f9bbbb3c000 x1873512753837568/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786724292 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3780.832133] Lustre: server umount lustre-OST0000 complete [ 3783.714271] Lustre: 16416:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724279/real 1786724279] req@ffff9f9bbbb3ea00 x1873512753837952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786724295 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3788.896777] Lustre: 16416:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724284/real 1786724284] req@ffff9f9a881f2680 x1873512753838208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786724300 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3792.083536] Lustre: server umount lustre-OST0001 complete [ 3812.588414] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 3817.172939] Key type lgssc unregistered [ 3817.591695] LNet: 105595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3817.604460] LNetError: 105595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3817.618763] LNet: Removed LNI 192.168.202.140@tcp [ 3818.915292] Key type .llcrypt unregistered [ 3818.918287] Key type ._llcrypt unregistered [ 3846.580082] Key type ._llcrypt registered [ 3846.584517] Key type .llcrypt registered [ 3846.738599] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 3861.166425] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3862.745418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3862.784289] alg: No test for adler32 (adler32-zlib) [ 3863.941473] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 3864.172640] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 3865.800202] Key type lgssc registered [ 3867.645995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3923.555607] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3936.023498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3936.071678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.322855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3937.347423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3937.416238] Lustre: lustre-MDT0000: new disk, initializing [ 3937.490107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3937.508272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3942.279662] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3954.877776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3955.010440] Lustre: 110053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3955.060666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3955.068654] Lustre: Skipped 1 previous similar message [ 3955.182306] Lustre: lustre-MDT0001: new disk, initializing [ 3955.263495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3955.309097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3955.318477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3959.710774] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3964.635434] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3973.779884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3974.011932] Lustre: lustre-OST0000: new disk, initializing [ 3974.017171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3974.023057] Lustre: 111991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3974.093829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3980.884841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3980.903330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3981.001976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3981.362781] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3996.762140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3996.953350] Lustre: lustre-OST0001: new disk, initializing [ 3996.955949] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3996.967776] Lustre: 113018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3997.058238] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4003.361284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4003.380527] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4003.487516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4004.615638] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4018.241364] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4026.599642] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4032.636765] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 12:22:23 (1786724543) === [ 4039.348956] Lustre: DEBUG MARKER: == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 12:22:30 (1786724550) [ 4039.627394] Lustre: 110088:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 4039.638322] Lustre: 110088:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4039.644939] Lustre: 110088:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4039.652824] Lustre: 110088:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 4039.665856] Lustre: 110088:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 4039.674933] Lustre: 110088:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4040.272856] Lustre: 110060:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 4040.287451] Lustre: 110060:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 9 previous similar messages [ 4040.296307] Lustre: 110060:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4040.304606] Lustre: 110060:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 4040.311381] Lustre: 110060:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4040.322514] Lustre: 110060:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 4040.327838] Lustre: 110060:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 7/369/2 [ 4040.335650] Lustre: 110060:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 4040.342859] Lustre: 110060:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 4040.347829] Lustre: 110060:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 4040.356421] Lustre: 110060:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4040.367539] Lustre: 110060:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 9 previous similar messages [ 4041.280141] Lustre: 110088:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 4041.286143] Lustre: 110088:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 137 previous similar messages [ 4041.302939] Lustre: 110060:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 4041.309771] Lustre: 110060:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 140 previous similar messages [ 4041.316479] Lustre: 110060:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4041.321710] Lustre: 110060:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 140 previous similar messages [ 4041.330566] Lustre: 110060:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 7/369/0 [ 4041.338974] Lustre: 110060:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 140 previous similar messages [ 4041.344897] Lustre: 110060:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 4041.348778] Lustre: 110060:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 140 previous similar messages [ 4041.365393] Lustre: 113804:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4041.370381] Lustre: 113804:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 143 previous similar messages [ 4042.978130] Lustre: *** cfs_fail_loc=1613, val=0*** [ 4053.239195] Lustre: DEBUG MARKER: == sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 12:22:44 (1786724564) [ 4054.446683] Lustre: 110061:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 4054.456169] Lustre: 110061:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 161 previous similar messages [ 4054.460565] Lustre: 110061:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 4054.468470] Lustre: 110061:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 158 previous similar messages [ 4054.474869] Lustre: 110061:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4054.481151] Lustre: 110061:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 158 previous similar messages [ 4054.490364] Lustre: 110061:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4054.497155] Lustre: 110061:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 158 previous similar messages [ 4054.507038] Lustre: 110061:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 4054.512643] Lustre: 110061:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 158 previous similar messages [ 4054.517535] Lustre: 110061:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4054.523294] Lustre: 110061:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 155 previous similar messages [ 4055.403320] Lustre: *** cfs_fail_loc=1614, val=0*** [ 4059.265564] Lustre: 111982:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 275, rollback = 2 [ 4059.275774] Lustre: 111982:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 11 previous similar messages [ 4059.283694] Lustre: 111982:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4059.294345] Lustre: 111982:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4059.311025] Lustre: 111982:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 3/275/0 [ 4059.318276] Lustre: 111982:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4059.327265] Lustre: 111982:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 1/4/0, quota 7/225/0 [ 4059.340472] Lustre: 111982:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4059.353947] Lustre: 111982:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 4059.367663] Lustre: 111982:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4059.376851] Lustre: 111982:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4059.395137] Lustre: 111982:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4066.544925] Lustre: DEBUG MARKER: == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 12:22:57 (1786724577) [ 4067.612770] Lustre: 113804:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 251 < left 306, rollback = 2 [ 4067.627563] Lustre: 113804:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 10 previous similar messages [ 4067.641756] Lustre: 113804:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4067.650902] Lustre: 113804:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 10 previous similar messages [ 4067.658118] Lustre: 113804:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/306/0 [ 4067.665560] Lustre: 113804:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 10 previous similar messages [ 4067.680949] Lustre: 113804:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 7/145/0, punch: 0/0/0, quota 1/3/0 [ 4067.692823] Lustre: 113804:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 10 previous similar messages [ 4067.701436] Lustre: 113804:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4067.711580] Lustre: 113804:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 10 previous similar messages [ 4067.718422] Lustre: 113804:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4067.727179] Lustre: 113804:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 10 previous similar messages [ 4069.728342] Lustre: *** cfs_fail_loc=1615, val=0*** [ 4069.732308] Lustre: Skipped 1 previous similar message [ 4088.988532] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_18b skipping excluded test 18b [ 4090.773868] Lustre: DEBUG MARKER: == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 12:23:21 (1786724601) [ 4091.087919] Lustre: 110088:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 4091.091916] Lustre: 110088:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 12 previous similar messages [ 4091.096086] Lustre: 110088:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 4091.101680] Lustre: 110088:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 12 previous similar messages [ 4091.105925] Lustre: 110088:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4091.109431] Lustre: 110088:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 12 previous similar messages [ 4091.117568] Lustre: 110088:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4091.121786] Lustre: 110088:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 12 previous similar messages [ 4091.125813] Lustre: 110088:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 4091.128992] Lustre: 110088:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 12 previous similar messages [ 4091.132503] Lustre: 110088:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4091.135641] Lustre: 110088:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 12 previous similar messages [ 4092.885879] Lustre: *** cfs_fail_loc=1617, val=0*** [ 4092.968657] Lustre: *** cfs_fail_loc=1617, val=0*** [ 4095.188104] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4095.195069] Lustre: Skipped 5 previous similar messages [ 4112.306179] Lustre: DEBUG MARKER: == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 12:23:43 (1786724623) [ 4114.490593] Lustre: *** cfs_fail_loc=1618, val=0*** [ 4114.494995] Lustre: Skipped 5 previous similar messages [ 4149.728902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4149.732899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.740545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4151.777270] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.779986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4151.801989] Lustre: Skipped 2 previous similar messages [ 4151.802636] Lustre: Skipped 2 previous similar messages [ 4154.923396] Lustre: server umount lustre-MDT0000 complete [ 4159.546890] LustreError: 110044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786724671 with bad export cookie 18130342737209346421 [ 4159.561363] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4159.566227] LustreError: 110044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4160.108800] Lustre: server umount lustre-MDT0001 complete [ 4175.866813] Lustre: server umount lustre-OST0000 complete [ 4177.190288] Lustre: 107214:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724673/real 1786724673] req@ffff9f9bb11e8000 x1873516299501184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786724689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4177.222679] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.903493] Lustre: server umount lustre-OST0001 complete [ 4196.735887] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 4206.627653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.063705] LustreError: 118731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4207.087075] LustreError: 118731:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4207.134140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.782440] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4212.194353] LustreError: 118732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4217.313984] LustreError: 118731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.187079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.516402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4224.203655] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4227.367672] Lustre: 119870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4234.437614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4241.256495] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4244.961718] LustreError: 120224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.978703] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:33) [ 4250.097022] LustreError: 120223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4252.173610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:116 to 0x280000401:161) [ 4252.454578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4252.723587] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4252.726351] Lustre: Skipped 1 previous similar message [ 4257.789405] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:33) [ 4257.796657] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 4262.014505] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4271.274601] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4274.773406] Lustre: 121742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4292.367699] Lustre: DEBUG MARKER: == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 12:26:43 (1786724803) [ 4292.714901] Lustre: 118728:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 4292.723393] Lustre: 118728:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 46 previous similar messages [ 4292.729159] Lustre: 118728:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4292.735702] Lustre: 118728:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4292.741029] Lustre: 118728:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4292.748216] Lustre: 118728:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4292.755062] Lustre: 118728:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 4292.761187] Lustre: 118728:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4292.766469] Lustre: 118728:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 4292.772856] Lustre: 118728:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4292.776268] Lustre: 118728:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4292.784372] Lustre: 118728:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4295.407357] Lustre: *** cfs_fail_loc=1618, val=0*** [ 4295.410792] Lustre: Skipped 3 previous similar messages [ 4332.001717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4332.027841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.059346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4332.075111] Lustre: Skipped 1 previous similar message [ 4334.563316] LustreError: 118732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.577639] LustreError: 118732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4335.354238] Lustre: server umount lustre-MDT0000 complete [ 4342.671427] LustreError: 118711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786724854 with bad export cookie 18130342737209361681 [ 4342.712835] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4343.056885] Lustre: server umount lustre-MDT0001 complete [ 4357.412224] Lustre: server umount lustre-OST0000 complete [ 4360.229238] Lustre: 107215:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786724856/real 1786724856] req@ffff9f9bb236ad80 x1873516299582336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786724872 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4360.266116] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.283211] Lustre: Skipped 3 previous similar messages [ 4362.034450] Lustre: server umount lustre-OST0001 complete [ 4382.717707] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 4397.326307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4397.932906] LustreError: 124310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.948200] LustreError: 124310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4398.059547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4403.798967] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4415.662029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.632706] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4425.094777] Lustre: 125451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4432.410558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4440.083372] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4441.832150] LustreError: 125803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4441.838298] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:65) [ 4441.843578] LustreError: 125803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4442.875133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4448.097761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4453.869267] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 4453.884272] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:65) [ 4454.734477] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4462.385737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4466.697355] Lustre: 127321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4473.311538] Lustre: 124311:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 252 < left 260, rollback = 2 [ 4473.331431] Lustre: 124311:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 21 previous similar messages [ 4473.341412] Lustre: 124311:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4473.353207] Lustre: 124311:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4473.375454] Lustre: 124311:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/260/0 [ 4473.391414] Lustre: 124311:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4473.405584] Lustre: 124311:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 0/0/0, quota 4/150/2 [ 4473.406362] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4473.417806] Lustre: 124311:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4473.417827] Lustre: 124311:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 1/17/2, delete: 0/0/0 [ 4473.417830] Lustre: 124311:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4473.417836] Lustre: 124311:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4473.417838] Lustre: 124311:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4473.480761] Lustre: Skipped 1 previous similar message [ 4501.411919] Lustre: DEBUG MARKER: == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 12:30:12 (1786725012) [ 4504.532942] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4504.537469] Lustre: Skipped 3 previous similar messages [ 4512.018687] Lustre: *** cfs_fail_loc=161c, val=0*** [ 4534.709279] Lustre: DEBUG MARKER: == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) ========================================================== 12:30:45 (1786725045) [ 4536.339818] Lustre: *** cfs_fail_loc=162e, val=0*** [ 4536.341535] Lustre: *** cfs_fail_loc=162e, val=0*** [ 4536.342736] Lustre: Skipped 7 previous similar messages [ 4550.061799] Lustre: DEBUG MARKER: == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 12:31:00 (1786725060) [ 4578.690166] Lustre: DEBUG MARKER: == sanity-lfsck test 19a: OST-object inconsistency self detect ========================================================== 12:31:28 (1786725088) [ 4596.273526] Lustre: DEBUG MARKER: == sanity-lfsck test 19b: OST-object inconsistency self repair ========================================================== 12:31:47 (1786725107) [ 4601.021994] Lustre: *** cfs_fail_loc=1611, val=0*** [ 4601.084697] Lustre: *** cfs_fail_loc=1611, val=0*** [ 4601.091390] Lustre: Skipped 3 previous similar messages [ 4607.188964] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.40@tcp inode [0x2000013a1:0x1f:0x0] object 0x280000401:203 extent [0-4095], client returned csum 0 (type 4), server csum 703fb3ad (type 4) [ 4608.274882] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.40@tcp inode [0x2000013a1:0x20:0x0] object 0x280000401:204 extent [0-4095], client returned csum 0 (type 4), server csum 7cbb935c (type 4) [ 4619.016853] Lustre: DEBUG MARKER: == sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================================================== 12:32:08 (1786725128) [ 4619.763368] Lustre: 124306:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 4619.771469] Lustre: 124306:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 104 previous similar messages [ 4619.777821] Lustre: 124306:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4619.782549] Lustre: 124306:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 4619.790439] Lustre: 124306:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4619.796661] Lustre: 124306:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 4619.802878] Lustre: 124306:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 4619.811424] Lustre: 124306:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 4619.818070] Lustre: 124306:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 4619.823532] Lustre: 124306:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 4619.830541] Lustre: 124306:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4619.836526] Lustre: 124306:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 104 previous similar messages [ 4623.832111] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4623.834136] Lustre: Skipped 3 previous similar messages [ 4653.111792] Lustre: DEBUG MARKER: == sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case ========================================================== 12:32:43 (1786725163) [ 4661.042663] Lustre: DEBUG MARKER: == sanity-lfsck test 21: run all LFSCK components by default ========================================================== 12:32:51 (1786725171) [ 4673.697590] Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) ========================================================== 12:33:04 (1786725184) [ 4676.056453] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4676.066243] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4676.068237] Lustre: Skipped 1 previous similar message [ 4693.136354] Lustre: DEBUG MARKER: == sanity-lfsck test 22b: LFSCK can repair unmatched pairs (2) ========================================================== 12:33:21 (1786725201) [ 4696.828944] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4696.844782] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4714.494874] Lustre: DEBUG MARKER: == sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) ========================================================== 12:33:44 (1786725224) [ 4716.866635] Lustre: *** cfs_fail_loc=1620, val=0*** [ 4732.893836] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_23b skipping excluded test 23b [ 4734.992957] Lustre: DEBUG MARKER: == sanity-lfsck test 23c: LFSCK can repair dangling name entry (3) ========================================================== 12:34:05 (1786725245) [ 4741.326113] Lustre: *** cfs_fail_loc=1621, val=127*** [ 4741.328263] Lustre: Skipped 1 previous similar message [ 4744.179556] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4766.294416] Lustre: DEBUG MARKER: == sanity-lfsck test 23d: LFSCK can repair a dangling name entry to a remote object ========================================================== 12:34:36 (1786725276) [ 4768.702390] Lustre: Failing over lustre-MDT0000 [ 4768.977355] Lustre: server umount lustre-MDT0000 complete [ 4769.063411] LustreError: 124305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4769.103222] LustreError: 124305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4769.249511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4769.262433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4779.165078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4779.245818] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4779.434599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4779.447664] Lustre: Skipped 3 previous similar messages [ 4779.476391] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4780.245644] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4784.637259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4784.686294] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4784.745031] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:130 to 0x2c0000401:161) [ 4784.746987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 4785.127813] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4787.192191] LustreError: 124306:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0000: [0x2000013a3:0x82:0x0] doesn't exist!: rc = -14 [ 4799.778032] Lustre: DEBUG MARKER: == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ========================================================== 12:35:10 (1786725310) [ 4802.150883] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4802.285250] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4802.293519] Lustre: Skipped 1 previous similar message [ 4815.931767] Lustre: DEBUG MARKER: == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ========================================================== 12:35:26 (1786725326) [ 4817.462528] Lustre: *** cfs_fail_loc=1623, val=0*** [ 4830.233286] Lustre: DEBUG MARKER: == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace ========================================================== 12:35:41 (1786725341) [ 4832.469241] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4843.698228] Lustre: DEBUG MARKER: == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========================================================== 12:35:54 (1786725354) [ 4858.240948] Lustre: DEBUG MARKER: == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan ========================================================== 12:36:08 (1786725368) [ 4860.147237] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4860.154578] Lustre: Skipped 1 previous similar message [ 4874.423741] Lustre: DEBUG MARKER: == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ========================================================== 12:36:24 (1786725384) [ 4890.765835] Lustre: DEBUG MARKER: == sanity-lfsck test 28: Skip the failed MDT(s) when handle orphan MDT-objects ========================================================== 12:36:41 (1786725401) [ 4891.259392] Lustre: 127744:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 4891.270840] Lustre: 127744:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 497 previous similar messages [ 4891.277242] Lustre: 127744:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4891.283664] Lustre: 127744:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4891.288075] Lustre: 127744:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4891.292696] Lustre: 127744:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4891.298163] Lustre: 127744:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 4891.303108] Lustre: 127744:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4891.306484] Lustre: 127744:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 4891.311995] Lustre: 127744:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4891.320575] Lustre: 127744:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4891.326097] Lustre: 127744:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 497 previous similar messages [ 4893.527405] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4893.535614] Lustre: Skipped 2 previous similar messages [ 4896.845844] Lustre: *** cfs_fail_loc=161c, val=0*** [ 4896.851464] Lustre: Skipped 1 previous similar message [ 4913.505046] Lustre: DEBUG MARKER: == sanity-lfsck test 29b: LFSCK can repair bad nlink count (2) ========================================================== 12:37:03 (1786725423) [ 4927.462173] Lustre: DEBUG MARKER: == sanity-lfsck test 29c: verify linkEA size limitation == 12:37:18 (1786725438) [ 4954.045180] Lustre: DEBUG MARKER: == sanity-lfsck test 29d: accessing non-existing inode shouldn't turn fs read-only (ldiskfs) ========================================================== 12:37:44 (1786725464) [ 4956.671206] LustreError: 124307:0:(osd_handler.c:284:osd_idc_find_or_init()) lustre-MDT0000: cannot lookup FID [0x2000013a3:0x9e:0x0]: rc = -2 [ 4963.689497] Lustre: DEBUG MARKER: == sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 12:37:54 (1786725474) [ 4965.441275] Lustre: *** cfs_fail_loc=161d, val=0*** [ 4965.449130] Lustre: Skipped 3 previous similar messages [ 4988.009255] Lustre: Failing over lustre-MDT0000 [ 4988.381558] Lustre: server umount lustre-MDT0000 complete [ 4989.410526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4989.414217] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.419059] LustreError: 124311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4989.419070] LustreError: 124311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4989.483268] Lustre: Skipped 6 previous similar messages [ 4998.618928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 4998.725099] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4998.913284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.949476] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5003.475180] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5004.258576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5004.259564] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5004.262734] Lustre: Skipped 3 previous similar messages [ 5004.289592] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5004.353979] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:166 to 0x2c0000401:193) [ 5004.354903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:295 to 0x280000401:321) [ 5016.984879] Lustre: DEBUG MARKER: == sanity-lfsck test 31a: The LFSCK can find/repair the name entry with bad name hash (1) ========================================================== 12:38:47 (1786725527) [ 5033.590946] Lustre: DEBUG MARKER: == sanity-lfsck test 31b: The LFSCK can find/repair the name entry with bad name hash (2) ========================================================== 12:39:04 (1786725544) [ 5052.477433] Lustre: DEBUG MARKER: == sanity-lfsck test 31c: Re-generate the lost master LMV EA for striped directory ========================================================== 12:39:22 (1786725562) [ 5054.180298] Lustre: *** cfs_fail_loc=1629, val=0*** [ 5054.184105] Lustre: Skipped 5 previous similar messages [ 5070.960841] Lustre: DEBUG MARKER: == sanity-lfsck test 31d: Set broken striped directory (modified after broken) as read-only ========================================================== 12:39:41 (1786725581) [ 5078.617603] Lustre: Failing over lustre-MDT0000 [ 5078.994815] Lustre: server umount lustre-MDT0000 complete [ 5081.059196] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5081.082540] Lustre: Skipped 2 previous similar messages [ 5088.004300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5088.106440] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5088.462355] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5092.833228] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5093.861595] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5093.877315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5093.891042] Lustre: Skipped 3 previous similar messages [ 5093.940725] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5093.996526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:295 to 0x280000401:353) [ 5093.997131] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:166 to 0x2c0000401:225) [ 5103.265595] Lustre: Failing over lustre-MDT0000 [ 5103.958598] Lustre: server umount lustre-MDT0000 complete [ 5104.098414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5112.528684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5112.708411] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5113.154143] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5116.114673] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5116.375279] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5118.442583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5118.456074] Lustre: Skipped 3 previous similar messages [ 5118.483689] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5118.559164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5118.561751] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:166 to 0x2c0000401:257) [ 5125.919186] Lustre: DEBUG MARKER: == sanity-lfsck test 31e: Re-generate the lost slave LMV EA for striped directory (1) ========================================================== 12:40:36 (1786725636) [ 5140.062630] Lustre: DEBUG MARKER: == sanity-lfsck test 31f: Re-generate the lost slave LMV EA for striped directory (2) ========================================================== 12:40:50 (1786725650) [ 5153.575300] Lustre: DEBUG MARKER: == sanity-lfsck test 31g: Repair the corrupted slave LMV EA ========================================================== 12:41:04 (1786725664) [ 5193.242570] Lustre: DEBUG MARKER: == sanity-lfsck test 31h: Repair the corrupted shard's name entry ========================================================== 12:41:43 (1786725703) [ 5195.557956] Lustre: *** cfs_fail_loc=162c, val=0*** [ 5195.565511] Lustre: Skipped 13 previous similar messages [ 5212.531478] Lustre: DEBUG MARKER: == sanity-lfsck test 32a: stop LFSCK when some OST failed ========================================================== 12:42:02 (1786725722) [ 5223.219303] LustreError: 148724:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5226.305107] LustreError: 148724:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 5226.312643] LustreError: 148724:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5226.893152] Lustre: Failing over lustre-OST0000 [ 5227.063421] Lustre: server umount lustre-OST0000 complete [ 5229.313089] LustreError: 148724:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 5229.330912] LustreError: 148724:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5230.088115] LustreError: 148724:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout interrupted [ 5230.098680] LustreError: lustre-OST0000-osc-MDT0000: operation lfsck_notify to node 0@lo failed: rc = -107 [ 5230.122885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5230.148497] Lustre: Skipped 5 previous similar messages [ 5242.546364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5242.784378] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5244.005748] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5244.064022] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5244.070142] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5244.088725] Lustre: Skipped 3 previous similar messages [ 5249.596140] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5259.959730] Lustre: DEBUG MARKER: == sanity-lfsck test 32b: stop LFSCK when some MDT failed ========================================================== 12:42:50 (1786725770) [ 5276.187783] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5295.028432] Lustre: 151528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5320.635841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5326.379499] Lustre: 152665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5344.613369] LustreError: 152803:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5347.616378] LustreError: 152803:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 5348.417900] Lustre: Failing over lustre-MDT0001 [ 5348.835270] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5348.847380] LustreError: Skipped 1 previous similar message [ 5348.851892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5350.385094] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5350.404809] Lustre: Skipped 1 previous similar message [ 5350.720147] LustreError: 152802:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 5350.753601] LustreError: 152802:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) Skipped 2 previous similar messages [ 5351.549629] Lustre: server umount lustre-MDT0001 complete [ 5355.489943] LustreError: 127744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5355.510045] LustreError: 127744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 34 previous similar messages [ 5370.596288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5370.848909] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5370.851819] Lustre: Skipped 1 previous similar message [ 5371.089914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5371.099701] Lustre: Skipped 3 previous similar messages [ 5371.142928] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5376.482429] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5376.497135] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5376.518332] Lustre: Skipped 1 previous similar message [ 5376.565908] Lustre: lustre-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5376.677335] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:97) [ 5376.681139] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:97) [ 5379.059668] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5389.144310] Lustre: DEBUG MARKER: == sanity-lfsck test 33: check LFSCK paramters =========== 12:44:59 (1786725899) [ 5406.412959] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5428.656526] Lustre: 155525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5454.350524] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5459.091104] Lustre: 156660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5461.744849] Lustre: 127088:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 5461.751182] Lustre: 127088:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1239 previous similar messages [ 5461.756386] Lustre: 127088:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 5461.760778] Lustre: 127088:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1239 previous similar messages [ 5461.766304] Lustre: 127088:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5461.771373] Lustre: 127088:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1239 previous similar messages [ 5461.778133] Lustre: 127088:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 5461.791668] Lustre: 127088:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1239 previous similar messages [ 5461.799670] Lustre: 127088:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 5461.809507] Lustre: 127088:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1239 previous similar messages [ 5461.822599] Lustre: 127088:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5461.827726] Lustre: 127088:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1239 previous similar messages [ 5480.214365] Lustre: DEBUG MARKER: == sanity-lfsck test 34: LFSCK can rebuild the lost agent object ========================================================== 12:46:30 (1786725990) [ 5481.916303] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_34 Only valid for ZFS backend [ 5484.180565] Lustre: DEBUG MARKER: == sanity-lfsck test 35: LFSCK can rebuild the lost agent entry ========================================================== 12:46:34 (1786725994) [ 5492.285454] Lustre: *** cfs_fail_loc=1631, val=0*** [ 5509.090933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5509.094237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5509.119697] Lustre: Skipped 4 previous similar messages [ 5509.134627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5511.769425] Lustre: server umount lustre-MDT0000 complete [ 5516.142578] LustreError: 124292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786726028 with bad export cookie 18130342737209434502 [ 5516.147656] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5516.156626] LustreError: 124292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5516.880840] Lustre: server umount lustre-MDT0001 complete [ 5532.157511] Lustre: server umount lustre-OST0000 complete [ 5535.202917] Lustre: 107213:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786726031/real 1786726031] req@ffff9f9bb1cffb80 x1873516301017344/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786726047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5535.941104] Lustre: server umount lustre-OST0001 complete [ 5554.808884] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5565.121487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5570.112521] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5579.121145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5583.538612] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5586.562245] Lustre: 160556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5595.128019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.268852] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5604.787155] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:129) [ 5610.929789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5611.004136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:540 to 0x280000401:577) [ 5616.619482] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:412 to 0x2c0000401:449) [ 5616.716807] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:129) [ 5617.350192] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5625.371223] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5629.654906] Lustre: 162427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5640.522635] Lustre: DEBUG MARKER: == sanity-lfsck test 36a: rebuild LOV EA for mirrored file (1) ========================================================== 12:49:11 (1786726151) [ 5642.158688] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36a needs >= 3 OSTs [ 5644.560667] Lustre: DEBUG MARKER: == sanity-lfsck test 36b: rebuild LOV EA for mirrored file (2) ========================================================== 12:49:15 (1786726155) [ 5646.268649] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36b needs >= 3 OSTs [ 5647.893098] Lustre: DEBUG MARKER: == sanity-lfsck test 36c: rebuild LOV EA for mirrored file (3) ========================================================== 12:49:18 (1786726158) [ 5649.500883] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36c needs >= 3 OSTs [ 5651.377302] Lustre: DEBUG MARKER: == sanity-lfsck test 37: LFSCK must skip a ORPHAN ======== 12:49:22 (1786726162) [ 5665.236637] Lustre: DEBUG MARKER: == sanity-lfsck test 38: LFSCK does not break foreign file and reverse is also true ========================================================== 12:49:35 (1786726175) [ 5684.699806] Lustre: DEBUG MARKER: == sanity-lfsck test 39: LFSCK does not break foreign dir and reverse is also true ========================================================== 12:49:55 (1786726195) [ 5701.062625] Lustre: DEBUG MARKER: == sanity-lfsck test 40a: LFSCK correctly fixes lmm_oi in composite layout ========================================================== 12:50:11 (1786726211) [ 5718.250882] Lustre: DEBUG MARKER: == sanity-lfsck test 41: SEL support in LFSCK ============ 12:50:28 (1786726228) [ 5738.836875] Lustre: DEBUG MARKER: == sanity-lfsck test 42: LFSCK repairs inconsistent MDT-object/OST-object encryption flags ========================================================== 12:50:49 (1786726249) [ 5776.288029] Lustre: *** cfs_fail_loc=1632, val=0*** [ 5793.040154] Lustre: DEBUG MARKER: == sanity-lfsck test 43: LFSCK does not loop endlessly on iget failure in scanning-phase1 ========================================================== 12:51:43 (1786726303) [ 5796.171278] Lustre: Failing over lustre-MDT0001 [ 5796.494702] Lustre: server umount lustre-MDT0001 complete [ 5799.907379] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5799.911734] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5799.937618] Lustre: Skipped 4 previous similar messages [ 5805.274781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5805.810615] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5805.814021] Lustre: lustre-MDT0001: Aborting client recovery [ 5805.814034] LustreError: 166216:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0001: Aborting recovery [ 5805.818595] LustreError: 166238:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -108 [ 5805.845919] Lustre: 166240:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5805.860687] Lustre: 166240:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 7f02c0ff-0e05-4ff3-a70c-857a2f238faa@ [ 5805.879397] Lustre: lustre-MDT0001: disconnecting 2 stale clients [ 5805.899333] Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] [ 5805.924215] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] [ 5806.025289] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5806.025779] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:161) [ 5811.185044] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5811.221401] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5811.224095] Lustre: Skipped 2 previous similar messages [ 5811.796341] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5816.006462] Lustre: *** cfs_fail_loc=1e2, val=483*** [ 5820.948905] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid [ 5821.300343] Lustre: DEBUG MARKER: osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid in FULL state after 0 sec [ 5829.987709] Lustre: DEBUG MARKER: == sanity-lfsck test 44: umount while lfsck is stopping == 12:52:20 (1786726340) [ 5839.113598] Lustre: *** cfs_fail_loc=1600, val=3*** [ 5842.528786] Lustre: Failing over lustre-MDT0000 [ 5842.837928] Lustre: server umount lustre-MDT0000 complete [ 5853.543144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.776111] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5854.172518] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5854.178116] Lustre: Skipped 2 previous similar messages [ 5855.466914] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5859.309781] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5859.317744] Lustre: Skipped 1 previous similar message [ 5859.345680] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5859.412970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:621 to 0x280000401:641) [ 5859.413038] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 5859.582184] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5871.139766] Lustre: DEBUG MARKER: == sanity-lfsck test 45: LFSCK should fix UID/GID/PROJID of OST object ========================================================== 12:53:01 (1786726381) [ 5898.700568] Lustre: Failing over lustre-OST0000 [ 5898.825420] Lustre: server umount lustre-OST0000 complete [ 5900.261393] LustreError: 163137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.280678] LustreError: 163137:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 34 previous similar messages [ 5905.228913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5917.018743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5917.215957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5917.224163] Lustre: Skipped 6 previous similar messages [ 5918.502296] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5919.230319] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5923.705432] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5929.380283] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5929.552526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5935.187455] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5935.481809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5942.236028] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93e087dca000.ost_server_uuid 50 [ 5944.342897] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93e087dca000.ost_server_uuid in FULL state after 0 sec [ 5993.956713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5993.961512] Lustre: Skipped 4 previous similar messages [ 5998.798366] Lustre: server umount lustre-MDT0000 complete [ 6008.041192] LustreError: 159397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786726520 with bad export cookie 18130342737209517095 [ 6008.058288] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6008.072308] LustreError: 159397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6008.867612] Lustre: server umount lustre-MDT0001 complete [ 6025.701935] Lustre: 107212:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786726521/real 1786726521] req@ffff9f9bc134bb80 x1873516301401728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786726537 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6029.792210] Lustre: 107215:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786726525/real 1786726525] req@ffff9f9a86435f80 x1873516301402112/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786726541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6034.301091] Lustre: server umount lustre-OST0000 complete [ 6035.028147] Lustre: 107215:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786726530/real 1786726530] req@ffff9f9bafd14380 x1873516301402752/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786726546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6035.042520] Lustre: 107215:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6040.038135] Lustre: 107212:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786726535/real 1786726535] req@ffff9f9bafd14000 x1873516301403136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786726551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6046.391857] Lustre: server umount lustre-OST0001 complete [ 6067.728711] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 6071.394591] Key type lgssc unregistered [ 6071.721315] LNet: 175353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6071.729583] LNetError: 175353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6071.754644] LNet: Removed LNI 192.168.202.140@tcp [ 6072.504368] Key type .llcrypt unregistered [ 6072.506530] Key type ._llcrypt unregistered [ 6102.516575] Key type ._llcrypt registered [ 6102.519592] Key type .llcrypt registered [ 6102.638232] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 6122.396560] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 6124.186990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6124.231723] alg: No test for adler32 (adler32-zlib) [ 6125.430335] Lustre: Lustre: Build Version: 2.17.57_1_g689e325 [ 6125.657088] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 6127.344508] Key type lgssc registered [ 6129.371696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6188.898537] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 6205.455882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6205.495177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6206.934062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6206.973695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6207.092280] Lustre: lustre-MDT0000: new disk, initializing [ 6207.252079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6207.303275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6213.870766] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6231.307528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6231.436757] Lustre: 179814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6231.473196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6231.476696] Lustre: Skipped 1 previous similar message [ 6231.536430] Lustre: lustre-MDT0001: new disk, initializing [ 6231.623068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6231.678626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6231.689217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6237.252680] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6242.439647] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6252.496289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6252.825184] Lustre: lustre-OST0000: new disk, initializing [ 6252.828427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6252.837782] Lustre: 181751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6252.930042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.082729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6255.091980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6255.224028] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6259.850976] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6271.835865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.978737] Lustre: lustre-OST0001: new disk, initializing [ 6271.984784] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6271.989989] Lustre: 182779:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6272.060599] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6278.733724] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6280.717241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6280.722599] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6280.791298] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6290.983566] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6299.792166] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6306.394836] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 13:00:17 (1786726817) === [ 6308.351195] Lustre: DEBUG MARKER: == sanity-lfsck test complete, duration 5983 sec ========= 13:00:18 (1786726818) [ 6310.287672] Lustre: DEBUG MARKER: === sanity-lfsck: start cleanup 13:00:20 (1786726820) === [ 6313.994315] Lustre: DEBUG MARKER: === sanity-lfsck: finish cleanup 13:00:24 (1786726824) === [ 6321.633319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6321.637330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6321.659345] Lustre: Skipped 1 previous similar message [ 6321.674050] Lustre: Skipped 3 previous similar messages [ 6325.462373] Lustre: server umount lustre-MDT0000 complete [ 6331.883769] LustreError: 179819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6331.916415] LustreError: 179819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6333.584142] LustreError: 181750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786726845 with bad export cookie 13840138108708497304 [ 6333.585118] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.590788] LustreError: 181750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6333.825320] Lustre: server umount lustre-MDT0001 complete [ 6352.003420] Lustre: server umount lustre-OST0000 complete [ 6352.232616] Lustre: 176971:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786726848/real 1786726848] req@ffff9f9bafd19880 x1873518670203136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786726864 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6352.267177] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.286901] Lustre: Skipped 2 previous similar messages [ 6355.168544] Lustre: 176969:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786726851/real 1786726851] req@ffff9f9bafd1b800 x1873518670203392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786726867 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6357.988743] Lustre: 176972:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786726853/real 1786726853] req@ffff9f9bbcc03100 x1873518670203648/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786726869 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6361.460520] Lustre: server umount lustre-OST0001 complete [ 6381.051369] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 6383.368084] Key type lgssc unregistered [ 6383.629981] LNet: 186267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6383.636595] LNetError: 186267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6383.656565] LNet: Removed LNI 192.168.202.140@tcp [ 6384.434180] Key type .llcrypt unregistered [ 6384.435697] Key type ._llcrypt unregistered