[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478076738 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002235] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.006743] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009107] LSM: Security Framework initializing [ 0.010057] Yama: becoming mindful. [ 0.011034] SELinux: Initializing. [ 0.012000] *** VALIDATE selinux *** [ 0.019185] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024071] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028082] *** VALIDATE tmpfs *** [ 0.029471] *** VALIDATE proc *** [ 0.031037] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036028] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040803] debug: unmapping init [mem 0xffffffffb6659000-0xffffffffb6660fff] [ 0.042180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043553] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045010] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049015] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051258] rcu: Hierarchical SRCU implementation. [ 0.053448] smp: Bringing up secondary CPUs ... [ 0.054579] x86: Booting SMP configuration: [ 0.055033] .... node #0, CPUs: #1 #2 #3 [ 0.058385] smp: Brought up 1 node, 4 CPUs [ 0.060015] smpboot: Max logical packages: 1 [ 0.061022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136885] node 0 deferred pages initialised in 73ms [ 0.139112] devtmpfs: initialized [ 0.140320] x86/mm: Memory block size: 128MB [ 0.143778] gcov: version magic: 0x41383552 [ 0.146163] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151291] pinctrl core: initialized pinctrl subsystem [ 0.153201] [ 0.153703] ************************************************************* [ 0.156015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158012] ** ** [ 0.160015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162014] ** ** [ 0.165014] ** This means that this kernel is built to expose internal ** [ 0.167014] ** IOMMU data structures, which may compromise security on ** [ 0.169013] ** your system. ** [ 0.171012] ** ** [ 0.173012] ** If you see this message and you are not debugging the ** [ 0.175012] ** kernel, report this immediately to your vendor! ** [ 0.177013] ** ** [ 0.179013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182013] ************************************************************* [ 0.184489] NET: Registered protocol family 16 [ 0.186441] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.195015] cpuidle: using governor menu [ 0.196787] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198515] PCI: Using configuration type 1 for base access [ 0.201138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.207279] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208086] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212009] cryptd: max_cpu_qlen set to 1000 [ 0.214201] ACPI: Added _OSI(Module Device) [ 0.215008] ACPI: Added _OSI(Processor Device) [ 0.215809] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.217012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.222631] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.229464] ACPI: Interpreter enabled [ 0.231060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.233012] ACPI: Using IOAPIC for interrupt routing [ 0.234104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239226] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.250805] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.252133] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.257092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.261054] acpiphp: Slot [2] registered [ 0.262037] acpiphp: Slot [5] registered [ 0.262934] acpiphp: Slot [6] registered [ 0.264059] acpiphp: Slot [7] registered [ 0.265031] acpiphp: Slot [8] registered [ 0.265934] acpiphp: Slot [9] registered [ 0.266134] acpiphp: Slot [10] registered [ 0.267020] acpiphp: Slot [3] registered [ 0.267806] acpiphp: Slot [4] registered [ 0.269057] acpiphp: Slot [11] registered [ 0.269907] acpiphp: Slot [12] registered [ 0.271050] acpiphp: Slot [13] registered [ 0.271955] acpiphp: Slot [14] registered [ 0.273083] acpiphp: Slot [15] registered [ 0.274103] acpiphp: Slot [16] registered [ 0.276100] acpiphp: Slot [17] registered [ 0.277080] acpiphp: Slot [18] registered [ 0.278084] acpiphp: Slot [19] registered [ 0.280139] acpiphp: Slot [20] registered [ 0.281113] acpiphp: Slot [21] registered [ 0.283236] acpiphp: Slot [22] registered [ 0.284127] acpiphp: Slot [23] registered [ 0.285098] acpiphp: Slot [24] registered [ 0.287099] acpiphp: Slot [25] registered [ 0.288105] acpiphp: Slot [26] registered [ 0.290100] acpiphp: Slot [27] registered [ 0.291055] acpiphp: Slot [28] registered [ 0.292013] acpiphp: Slot [29] registered [ 0.293076] acpiphp: Slot [30] registered [ 0.294085] acpiphp: Slot [31] registered [ 0.296065] PCI host bridge to bus 0000:00 [ 0.297018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.299022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.302022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.304024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.306020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.308041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.309136] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.310000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.311581] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.326715] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.329019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.331014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.335489] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.337980] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.346070] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.352017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.369022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.375017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.380625] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.389019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.394022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.408029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.416954] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.423017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.428016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.442017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.452506] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.457012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.461013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.477017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.484109] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.490011] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.495019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.506013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.517539] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.524019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.530016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.545047] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.553790] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.559017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.565016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.577016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.587188] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.590290] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.592391] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.594370] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.597210] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.601122] iommu: Default domain type: Passthrough [ 0.603428] SCSI subsystem initialized [ 0.605303] ACPI: bus type USB registered [ 0.606118] usbcore: registered new interface driver usbfs [ 0.608084] usbcore: registered new interface driver hub [ 0.610085] usbcore: registered new device driver usb [ 0.611157] pps_core: LinuxPPS API ver. 1 registered [ 0.613016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.615072] PTP clock support registered [ 0.618047] EDAC MC: Ver: 3.0.0 [ 0.620163] PCI: Using ACPI for IRQ routing [ 0.621771] NetLabel: Initializing [ 0.622008] NetLabel: domain hash size = 128 [ 0.623014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.625092] NetLabel: unlabeled traffic allowed by default [ 0.627120] vgaarb: loaded [ 0.628333] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.629018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.634688] clocksource: Switched to clocksource kvm-clock [ 0.750599] VFS: Disk quotas dquot_6.6.0 [ 0.752158] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.754780] *** VALIDATE ramfs *** [ 0.756093] *** VALIDATE hugetlbfs *** [ 0.757783] pnp: PnP ACPI init [ 0.760253] pnp: PnP ACPI: found 6 devices [ 0.774893] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.779040] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.780789] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.781924] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.783317] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.784798] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.787418] NET: Registered protocol family 2 [ 0.789819] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.793688] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.797279] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.802884] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.806110] TCP: Hash tables configured (established 65536 bind 65536) [ 0.808581] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.811820] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.814312] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.816292] NET: Registered protocol family 1 [ 0.818476] RPC: Registered named UNIX socket transport module. [ 0.819766] RPC: Registered udp transport module. [ 0.820693] RPC: Registered tcp transport module. [ 0.821612] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.823390] NET: Registered protocol family 44 [ 0.825172] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.827640] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.829995] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.832541] PCI: CLS 0 bytes, default 64 [ 0.834263] Unpacking initramfs... [ 2.240200] debug: unmapping init [mem 0xffff8a757cc54000-0xffff8a757ffbffff] [ 2.247370] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.250009] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.254776] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.853656] Initialise system trusted keyrings [ 2.856418] Key type blacklist registered [ 2.858374] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.875668] zbud: loaded [ 2.879086] *** VALIDATE nfs *** [ 2.880607] *** VALIDATE nfs4 *** [ 2.882510] pstore: using deflate compression [ 2.886680] Platform Keyring initialized [ 2.996329] NET: Registered protocol family 38 [ 2.999123] Key type asymmetric registered [ 3.000507] Asymmetric key parser 'x509' registered [ 3.002015] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.004117] io scheduler mq-deadline registered [ 3.005265] io scheduler kyber registered [ 3.006465] io scheduler bfq registered [ 3.007724] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.010453] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.012764] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.015766] ACPI: Power Button [PWRF] [ 3.019781] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.025448] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.037578] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.045528] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.056934] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.083720] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.111163] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.115733] Non-volatile memory driver v1.3 [ 3.117325] Linux agpgart interface v0.103 [ 3.145796] virtio_blk virtio1: [vda] 145808 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.148880] vda: detected capacity change from 0 to 74653696 [ 3.163539] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.166695] vdb: detected capacity change from 0 to 1073741824 [ 3.184611] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.186871] vdc: detected capacity change from 0 to 2621440000 [ 3.204094] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.207263] vdd: detected capacity change from 0 to 2621440000 [ 3.223389] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.225775] vde: detected capacity change from 0 to 4294967296 [ 3.242371] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.245047] vdf: detected capacity change from 0 to 4294967296 [ 3.261936] libphy: Fixed MDIO Bus: probed [ 3.272545] usbcore: registered new interface driver usbserial_generic [ 3.276312] usbserial: USB Serial support registered for generic [ 3.278566] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.282876] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.284625] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.287479] mousedev: PS/2 mouse device common for all mice [ 3.290249] rtc_cmos 00:05: RTC can wake from S4 [ 3.292751] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.296160] rtc_cmos 00:05: registered as rtc0 [ 3.297434] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.299666] intel_pstate: CPU model not supported [ 3.302131] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.303929] hid: raw HID events driver (C) Jiri Kosina [ 3.309375] usbcore: registered new interface driver usbhid [ 3.309510] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.311477] usbhid: USB HID core driver [ 3.316494] drop_monitor: Initializing network drop monitor service [ 3.319307] Initializing XFRM netlink socket [ 3.321909] NET: Registered protocol family 10 [ 3.326174] Segment Routing with IPv6 [ 3.327967] NET: Registered protocol family 17 [ 3.330419] mpls_gso: MPLS GSO support [ 3.338162] RAS: Correctable Errors collector initialized. [ 3.340889] AVX version of gcm_enc/dec engaged. [ 3.342743] AES CTR mode by8 optimization enabled [ 3.435333] sched_clock: Marking stable (3435295076, 0)->(4274592918, -839297842) [ 3.439479] registered taskstats version 1 [ 3.441532] Loading compiled-in X.509 certificates [ 3.443496] zswap: loaded using pool lzo/zbud [ 3.465781] Key type big_key registered [ 3.479376] Key type encrypted registered [ 3.481299] ima: No TPM chip found, activating TPM-bypass! [ 3.483549] ima: Allocated hash algorithm: sha1 [ 3.485232] ima: No architecture policies found [ 3.487318] evm: Initialising EVM extended attributes: [ 3.489010] evm: security.selinux [ 3.490115] evm: security.ima [ 3.491612] evm: security.capability [ 3.492845] evm: HMAC attrs: 0x1 [ 3.495682] rtc_cmos 00:05: setting system clock to 2026-07-13 12:38:36 UTC (1783946316) [ 3.503651] debug: unmapping init [mem 0xffffffffb7603000-0xffffffffb77fffff] [ 3.506822] debug: unmapping init [mem 0xffffffffb6382000-0xffffffffb6658fff] [ 3.516133] Write protecting the kernel read-only data: 28672k [ 3.520106] debug: unmapping init [mem 0xffffffffb4a03000-0xffffffffb4bfffff] [ 3.523254] debug: unmapping init [mem 0xffffffffb5314000-0xffffffffb53fffff] [ 3.567355] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.575667] systemd[1]: Detected virtualization kvm. [ 3.577566] systemd[1]: Detected architecture x86-64. [ 3.579452] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.609051] systemd[1]: No hostname configured. [ 3.610802] systemd[1]: Set hostname to . [ 3.612955] random: systemd: uninitialized urandom read (16 bytes read) [ 3.614936] systemd[1]: Initializing machine ID from random generator. [ 3.757841] random: systemd: uninitialized urandom read (16 bytes read) [ 3.760654] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.766156] random: systemd: uninitialized urandom read (16 bytes read) [ 3.768501] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.772720] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.497900] device-mapper: uevent: version 1.0.3 [ 4.500518] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [[ 5.133354] random: fast init done  OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.379130] virtio_net virtio0 ens2: renamed from eth0 [ 5.471702] scsi host0: ata_piix [ 5.492228] scsi host1: ata_piix [ 5.493813] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.496446] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.004956] random: crng init done [ 10.007289] random: 7 urandom warning(s) missed due to ratelimiting [ 10.020939] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.633302] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.930257] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.207990] SELinux: Disabled at runtime. [ 12.270234] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.280890] systemd[1]: Detected virtualization kvm. [ 12.283052] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.831315] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.834860] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.840904] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.844924] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.848594] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.858185] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.862903] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ 13.049259] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.525654] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.780026] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.839305] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.981800] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.997633] EDAC sbridge: Ver: 1.1.2 [ 15.827210] Key type dns_resolver registered [ 16.125520] NFS: Registering the id_resolver key type [ 16.127506] Key type id_resolver registered [ 16.129039] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg238-server login: [ 113.120807] hrtimer: interrupt took 8085164 ns [ 126.817376] libcfs: loading out-of-tree module taints kernel. [ 126.843295] Key type ._llcrypt registered [ 126.844416] Key type .llcrypt registered [ 126.905572] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 138.143236] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 138.963364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 138.971589] alg: No test for adler32 (adler32-zlib) [ 140.093319] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 140.809764] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 142.479402] Key type lgssc registered [ 144.479680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 161.532882] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 210.443634] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 221.093170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 221.130444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.412172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 222.439481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 222.534965] Lustre: lustre-MDT0000: new disk, initializing [ 222.599339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.610514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 226.379277] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.806179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 240.897703] Lustre: 6484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 240.923319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 240.927944] Lustre: Skipped 1 previous similar message [ 240.988371] Lustre: lustre-MDT0001: new disk, initializing [ 241.038538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.061400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 241.077878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 246.993546] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.283838] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 270.214482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 270.550363] Lustre: lustre-OST0000: new disk, initializing [ 270.556747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 270.575113] Lustre: 8424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 270.685805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 277.139965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 277.157453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 277.272183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 278.008336] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.841674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 294.003160] Lustre: lustre-OST0001: new disk, initializing [ 294.008850] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 294.019066] Lustre: 9497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 294.079966] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 300.080079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 300.088172] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 300.156217] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 301.592646] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 314.309735] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 321.840119] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 331.286694] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing check_logdir /tmp/testlogs/ [ 338.268856] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing yml_node [ 343.286759] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 346.528347] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 350.286539] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 352.185060] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Mon Jul 13 08:44:23 EDT 2026 [ 373.286635] Lustre: DEBUG MARKER: excepting tests: 18b 23b [ 385.376374] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 397.286421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.289583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 397.345659] Lustre: Skipped 1 previous similar message [ 397.376253] Lustre: Skipped 3 previous similar messages [ 402.404728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 402.411266] Lustre: Skipped 3 previous similar messages [ 402.572183] Lustre: server umount lustre-MDT0000 complete [ 412.643390] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.671750] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 416.433762] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783946729 with bad export cookie 16508878329404772229 [ 416.447907] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 416.449131] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 416.981153] Lustre: server umount lustre-MDT0001 complete [ 433.120352] Lustre: 3618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783946730/real 1783946730] req@ffff8a75ffa89f80 x1870603436015616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783946746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 433.142788] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 433.155112] Lustre: Skipped 2 previous similar messages [ 437.215311] Lustre: 3621:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783946734/real 1783946734] req@ffff8a75f5136a00 x1870603436015872/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783946750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 437.388575] Lustre: server umount lustre-OST0000 complete [ 439.008190] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783946735/real 1783946735] req@ffff8a75ffa88a80 x1870603436016128/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783946751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 442.402037] Lustre: 3621:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783946739/real 1783946739] req@ffff8a75e74c5180 x1870603436016512/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783946755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 446.664549] Lustre: server umount lustre-OST0001 complete [ 464.463810] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 468.041693] Key type lgssc unregistered [ 468.509917] LNet: 14775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.518449] LNetError: 14775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 468.551501] LNet: Removed LNI 192.168.202.138@tcp [ 469.947695] Key type .llcrypt unregistered [ 469.951403] Key type ._llcrypt unregistered [ 503.177334] Key type ._llcrypt registered [ 503.180416] Key type .llcrypt registered [ 503.375766] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 524.105953] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 526.202239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 526.371830] alg: No test for adler32 (adler32-zlib) [ 527.611664] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 527.916661] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 529.672540] Key type lgssc registered [ 531.189623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 621.901931] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 637.504874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 637.578743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.979930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 639.102060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 639.199569] Lustre: lustre-MDT0000: new disk, initializing [ 639.338576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 639.359836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 644.927701] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 660.494778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 660.634919] Lustre: 19249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 660.656469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 660.659311] Lustre: Skipped 1 previous similar message [ 660.708670] Lustre: lustre-MDT0001: new disk, initializing [ 660.781836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 660.812781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 660.831957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 665.774872] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 670.521748] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 682.228777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 682.536738] Lustre: lustre-OST0000: new disk, initializing [ 682.540960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 682.548771] Lustre: 21192:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 682.653607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 683.227755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 683.258955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 683.338603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 691.222202] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 708.301047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 708.424215] Lustre: lustre-OST0001: new disk, initializing [ 708.428791] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 708.436612] Lustre: 22215:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 708.482744] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 715.778622] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 718.405679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 718.414473] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 718.490269] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 729.152968] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 738.235486] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 747.759745] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 08:50:58 (1783947058) === [ 751.055278] Lustre: DEBUG MARKER: == sanity-lfsck test 0: Control LFSCK manually =========== 08:51:02 (1783947062) [ 751.198566] Lustre: 19254:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 751.205779] Lustre: 19254:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 751.210288] Lustre: 19254:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 751.219208] Lustre: 19254:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 751.221366] Lustre: 19254:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 751.233495] Lustre: 19254:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 751.778832] Lustre: 22996:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 751.788711] Lustre: 22996:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 14 previous similar messages [ 751.797574] Lustre: 22996:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 751.806708] Lustre: 22996:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 14 previous similar messages [ 751.813861] Lustre: 22996:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 751.824391] Lustre: 22996:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 14 previous similar messages [ 751.831964] Lustre: 22996:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 751.840194] Lustre: 22996:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 14 previous similar messages [ 751.844918] Lustre: 22996:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 751.854965] Lustre: 22996:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 14 previous similar messages [ 751.866248] Lustre: 22996:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 751.879191] Lustre: 22996:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 14 previous similar messages [ 752.794472] Lustre: 22996:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 752.813674] Lustre: 22996:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 47 previous similar messages [ 752.830779] Lustre: 22996:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 752.835165] Lustre: 22996:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 47 previous similar messages [ 752.840884] Lustre: 22996:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 752.847207] Lustre: 22996:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 47 previous similar messages [ 752.853863] Lustre: 22996:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 752.861996] Lustre: 22996:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 47 previous similar messages [ 752.869244] Lustre: 22996:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 752.874372] Lustre: 22996:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 47 previous similar messages [ 752.888026] Lustre: 22996:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 752.901288] Lustre: 22996:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 47 previous similar messages [ 754.806921] Lustre: 19253:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 754.817913] Lustre: 19253:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 98 previous similar messages [ 754.871676] Lustre: 22996:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 754.876900] Lustre: 22996:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 101 previous similar messages [ 754.879197] Lustre: 22996:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 754.892761] Lustre: 22996:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 101 previous similar messages [ 754.900043] Lustre: 22996:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 754.909104] Lustre: 22996:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 101 previous similar messages [ 754.913049] Lustre: 22996:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 754.915298] Lustre: 22996:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 101 previous similar messages [ 754.921670] Lustre: 22996:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.933739] Lustre: 22996:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 101 previous similar messages [ 759.707248] Lustre: *** cfs_fail_loc=1600, val=3*** [ 762.139893] Lustre: 23405:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0001: opcode 2: before 259 < left 274, rollback = 2 [ 762.148157] Lustre: 23405:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 116 previous similar messages [ 762.150620] Lustre: 23406:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 762.168651] Lustre: 23405:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 762.187107] Lustre: 23406:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 114 previous similar messages [ 762.187157] Lustre: 23406:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 762.187163] Lustre: 23406:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 113 previous similar messages [ 762.187172] Lustre: 23406:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 762.187179] Lustre: 23406:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 113 previous similar messages [ 762.187188] Lustre: 23406:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.187194] Lustre: 23406:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 113 previous similar messages [ 762.269981] Lustre: 23405:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 118 previous similar messages [ 762.783380] Lustre: *** cfs_fail_loc=1600, val=3*** [ 765.159359] Lustre: *** cfs_fail_loc=1600, val=3*** [ 778.356403] Lustre: 23606:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 274, rollback = 2 [ 778.378208] Lustre: 23606:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 63 previous similar messages [ 778.387784] Lustre: 23606:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 778.392107] Lustre: 23613:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 778.396914] Lustre: 23606:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 64 previous similar messages [ 778.396948] Lustre: 23606:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 778.396952] Lustre: 23606:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 64 previous similar messages [ 778.396957] Lustre: 23606:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 778.396960] Lustre: 23606:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 64 previous similar messages [ 778.396965] Lustre: 23606:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 778.396969] Lustre: 23606:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 64 previous similar messages [ 778.457818] Lustre: 23613:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 61 previous similar messages [ 783.841242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 783.850776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 783.872828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 785.377785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 785.380397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 785.387394] Lustre: Skipped 1 previous similar message [ 785.391042] Lustre: Skipped 1 previous similar message [ 787.479268] Lustre: server umount lustre-MDT0000 complete [ 791.874873] LustreError: 19239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947104 with bad export cookie 12563524709305191857 [ 791.885870] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 791.914634] LustreError: 19239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 792.627620] Lustre: server umount lustre-MDT0001 complete [ 807.216896] Lustre: server umount lustre-OST0000 complete [ 811.039146] Lustre: 16391:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783947108/real 1783947108] req@ffff8a75c12aa680 x1870603841817472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783947124 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 811.071193] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 811.092249] Lustre: Skipped 1 previous similar message [ 811.655710] Lustre: server umount lustre-OST0001 complete [ 821.175466] Lustre: DEBUG MARKER: == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 08:52:12 (1783947132) [ 836.271185] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 848.930735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.476334] LustreError: 26277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.505808] LustreError: 26277:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 849.628833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 854.961088] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 855.031492] LustreError: 26278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 859.178183] LustreError: 26277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 864.239653] LustreError: 26278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 865.033763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.416312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 870.639248] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 874.498517] Lustre: 27417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 882.612512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 883.073263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 891.259200] LustreError: 27771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 891.866713] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 894.330112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 899.580324] LustreError: 27771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 899.598189] LustreError: 27771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 904.168478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 909.816155] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:40 to 0x2c0000401:65) [ 912.693183] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 921.642768] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 925.990312] Lustre: 29290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 927.719815] Lustre: 29105:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 927.723700] Lustre: 29105:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 2 previous similar messages [ 927.726752] Lustre: 29105:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 927.730482] Lustre: 29105:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 927.734562] Lustre: 29105:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 927.738850] Lustre: 29105:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 927.742864] Lustre: 29105:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 927.749810] Lustre: 29105:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 927.755665] Lustre: 29105:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 927.760464] Lustre: 29105:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 927.766750] Lustre: 29105:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 933.118881] Lustre: *** cfs_fail_loc=1501, val=0*** [ 943.368261] Lustre: Failing over lustre-MDT0000 [ 943.894172] Lustre: server umount lustre-MDT0000 complete [ 945.632646] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 945.634724] LustreError: 29105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 945.649105] Lustre: Skipped 2 previous similar messages [ 945.682935] LustreError: 29105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 956.483288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 956.691168] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 957.068889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 957.071356] Lustre: Skipped 1 previous similar message [ 957.124897] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 962.546403] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 962.575700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 962.623151] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 962.657993] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 962.658898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 964.139766] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 968.227800] Lustre: *** cfs_fail_loc=1505, val=0*** [ 977.099854] Lustre: DEBUG MARKER: == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 08:54:48 (1783947288) [ 978.872285] Lustre: 26272:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 978.879391] Lustre: 26272:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 322 previous similar messages [ 978.884580] Lustre: 26272:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 978.894621] Lustre: 26272:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 978.905960] Lustre: 26272:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 978.914752] Lustre: 26272:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 978.923148] Lustre: 26272:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 978.929075] Lustre: 26272:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 978.933989] Lustre: 26272:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 978.939923] Lustre: 26272:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 978.945670] Lustre: 26272:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 978.953176] Lustre: 26272:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 983.761679] Lustre: *** cfs_fail_loc=1502, val=0*** [ 994.817414] Lustre: Failing over lustre-MDT0000 [ 995.151230] Lustre: server umount lustre-MDT0000 complete [ 998.369958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 998.385500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.409597] Lustre: Skipped 1 previous similar message [ 998.420516] LustreError: 29303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.443767] LustreError: 29303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 1006.703531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1006.955918] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1007.376155] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1012.442898] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1012.716573] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1012.717146] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1012.740981] Lustre: Skipped 3 previous similar messages [ 1012.760268] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1012.825982] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:169 to 0x2c0000401:193) [ 1012.825987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 1016.121714] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1024.682878] Lustre: DEBUG MARKER: == sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 08:55:35 (1783947335) [ 1031.917111] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1031.932721] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1031.939885] Lustre: Skipped 1 previous similar message [ 1041.059523] Lustre: Failing over lustre-MDT0000 [ 1041.353423] Lustre: server umount lustre-MDT0000 complete [ 1043.426714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.444957] Lustre: Skipped 4 previous similar messages [ 1054.243818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1054.388520] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1054.644789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1054.647602] Lustre: Skipped 1 previous similar message [ 1054.715041] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1059.814656] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1059.827032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1059.834536] Lustre: Skipped 3 previous similar messages [ 1059.873367] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1059.949494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:233 to 0x280000401:257) [ 1059.949657] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:257) [ 1060.688729] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1064.516069] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1072.398540] Lustre: DEBUG MARKER: == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 08:56:23 (1783947383) [ 1073.800236] Lustre: 26274:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 256 < left 278, rollback = 2 [ 1073.809698] Lustre: 26274:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 644 previous similar messages [ 1073.813529] Lustre: 26274:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 1073.820956] Lustre: 26274:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 1073.830291] Lustre: 26274:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1073.836183] Lustre: 26274:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 1073.841654] Lustre: 26274:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1073.848439] Lustre: 26274:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 1073.855362] Lustre: 26274:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 1073.859782] Lustre: 26274:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 1073.871159] Lustre: 26274:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1073.875859] Lustre: 26274:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 1079.813708] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1089.328306] Lustre: Failing over lustre-MDT0000 [ 1090.529369] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.530065] LustreError: 26272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1090.531836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1090.565511] Lustre: Skipped 5 previous similar messages [ 1090.592807] LustreError: 26272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 1091.745832] Lustre: server umount lustre-MDT0000 complete [ 1104.255952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1104.437856] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1104.792854] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1109.984364] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1110.009374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1110.022451] Lustre: Skipped 3 previous similar messages [ 1110.053983] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1110.134380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:297 to 0x280000401:321) [ 1110.137071] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:297 to 0x2c0000401:321) [ 1110.485588] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1122.144047] Lustre: DEBUG MARKER: == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 08:57:13 (1783947433) [ 1129.559108] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1139.454211] Lustre: Failing over lustre-MDT0000 [ 1140.704150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1140.706400] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.748166] Lustre: Skipped 3 previous similar messages [ 1141.754666] Lustre: server umount lustre-MDT0000 complete [ 1155.742702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1155.884163] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1156.194166] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1161.187001] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1161.187434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1161.223017] Lustre: Skipped 3 previous similar messages [ 1161.245676] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1161.316060] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:361 to 0x2c0000401:385) [ 1161.318518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:361 to 0x280000401:385) [ 1161.741449] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1172.895530] Lustre: DEBUG MARKER: == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 08:58:03 (1783947483) [ 1182.969661] Lustre: *** cfs_fail_loc=1605, val=0*** [ 1193.679935] Lustre: Failing over lustre-MDT0000 [ 1195.925680] Lustre: server umount lustre-MDT0000 complete [ 1197.027425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1207.187091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1207.341619] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1207.621915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.631571] Lustre: Skipped 2 previous similar messages [ 1207.665558] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1212.903238] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1212.920066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1212.924859] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1212.930378] Lustre: Skipped 3 previous similar messages [ 1212.973666] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1213.018225] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:425 to 0x2c0000401:449) [ 1213.021736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:425 to 0x280000401:449) [ 1222.472765] Lustre: DEBUG MARKER: == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 08:58:53 (1783947533) [ 1223.766823] Lustre: 26274:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1223.776383] Lustre: 26274:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 968 previous similar messages [ 1223.781686] Lustre: 26274:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1223.785445] Lustre: 26274:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1223.788766] Lustre: 26274:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1223.792070] Lustre: 26274:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1223.796525] Lustre: 26274:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1223.800422] Lustre: 26274:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1223.804056] Lustre: 26274:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1223.807146] Lustre: 26274:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1223.810907] Lustre: 26274:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1223.813894] Lustre: 26274:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1228.135713] Lustre: *** cfs_fail_loc=161d, val=0*** [ 1235.935943] Lustre: Failing over lustre-MDT0000 [ 1236.225747] Lustre: server umount lustre-MDT0000 complete [ 1238.501435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.510758] LustreError: 26273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1238.523461] Lustre: Skipped 5 previous similar messages [ 1238.565908] LustreError: 26273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 1247.044367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1247.211373] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.494686] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1252.834453] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1252.862147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1252.874497] Lustre: Skipped 3 previous similar messages [ 1252.914455] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1252.980284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:489 to 0x280000401:513) [ 1252.982803] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 1253.648501] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1262.888275] Lustre: DEBUG MARKER: == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 08:59:34 (1783947574) [ 1265.829373] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1276.513967] Lustre: DEBUG MARKER: == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 08:59:47 (1783947587) [ 1283.043683] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1284.292384] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1296.497567] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 09:00:07 (1783947607) [ 1332.219568] Lustre: Failing over lustre-MDT0000 [ 1334.463794] Lustre: server umount lustre-MDT0000 complete [ 1334.759573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1340.477417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1350.117623] Lustre: 16389:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783947647/real 1783947647] req@ffff8a74c998d180 x1870603842444160/t0(0) o400->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783947663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1350.162858] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1351.820872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1365.858376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1365.881734] Lustre: lustre-MDT0000: reset Object Index mappings [ 1376.158757] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1381.362590] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1381.388309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1381.409434] Lustre: Skipped 3 previous similar messages [ 1381.448317] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1381.498765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:609) [ 1381.500937] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:609) [ 1381.809069] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1386.155613] LustreError: 42948:0:(lfsck_engine.c:1018:lfsck_master_engine()) lustre-MDT0000-osd: master engine fail to verify the .lustre/lost+found/, go ahead: rc = -115 [ 1386.179336] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1387.232116] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1388.255376] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1390.307036] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1390.314502] Lustre: Skipped 1 previous similar message [ 1396.066382] Lustre: Failing over lustre-MDT0000 [ 1396.527741] Lustre: server umount lustre-MDT0000 complete [ 1396.716095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.739776] Lustre: Skipped 8 previous similar messages [ 1407.860639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1413.548736] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1413.688102] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:641) [ 1413.688967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:641) [ 1417.409199] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1426.821289] Lustre: DEBUG MARKER: == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 09:02:17 (1783947737) [ 1429.496955] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1439.027865] Lustre: Failing over lustre-MDT0000 [ 1439.204464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1439.240570] Lustre: server umount lustre-MDT0000 complete [ 1444.783918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1457.607282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1460.704176] Lustre: 16391:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783947757/real 1783947757] req@ffff8a74ca980e00 x1870603842547200/t0(0) o400->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783947773 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1470.062983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1470.095669] Lustre: lustre-MDT0000: reset Object Index mappings [ 1470.947219] LustreError: 16388:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a7601cdfb80 x1870603842554752/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1471.369567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.379843] Lustre: Skipped 3 previous similar messages [ 1471.444689] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1471.456046] Lustre: Skipped 1 previous similar message [ 1476.583116] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1476.589750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1476.593734] Lustre: Skipped 1 previous similar message [ 1476.619316] Lustre: Skipped 7 previous similar messages [ 1476.640572] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1476.654541] Lustre: Skipped 1 previous similar message [ 1476.731562] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:705) [ 1476.733312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:705) [ 1477.004725] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1481.348959] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1495.049858] Lustre: Failing over lustre-MDT0000 [ 1495.604342] Lustre: server umount lustre-MDT0000 complete [ 1497.056464] LustreError: 29303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.087483] LustreError: 29303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 87 previous similar messages [ 1507.136136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1507.366912] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1507.374739] LustreError: Skipped 2 previous similar messages [ 1513.023244] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:737) [ 1513.026658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:737) [ 1513.356864] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1517.840181] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1517.842226] Lustre: Skipped 84 previous similar messages [ 1526.778407] Lustre: DEBUG MARKER: == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 09:03:58 (1783947838) [ 1528.145201] Lustre: 26274:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1528.152932] Lustre: 26274:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1258 previous similar messages [ 1528.158631] Lustre: 26274:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1528.174628] Lustre: 26274:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1258 previous similar messages [ 1528.182595] Lustre: 26274:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1528.188488] Lustre: 26274:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1258 previous similar messages [ 1528.196159] Lustre: 26274:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1528.203912] Lustre: 26274:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1258 previous similar messages [ 1528.208470] Lustre: 26274:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1528.212948] Lustre: 26274:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1258 previous similar messages [ 1528.216993] Lustre: 26274:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1528.221630] Lustre: 26274:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1258 previous similar messages [ 1534.787509] Lustre: *** cfs_fail_loc=1600, val=1*** [ 1534.793032] Lustre: Skipped 7 previous similar messages [ 1555.804470] Lustre: DEBUG MARKER: == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 09:04:26 (1783947866) [ 1563.427768] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1563.433595] Lustre: Skipped 8 previous similar messages [ 1588.936948] Lustre: DEBUG MARKER: == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 09:05:00 (1783947900) [ 1600.424168] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1600.441533] Lustre: Skipped 12 previous similar messages [ 1606.782142] Lustre: Failing over lustre-MDT0000 [ 1607.075499] Lustre: server umount lustre-MDT0000 complete [ 1610.227742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1618.315279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1618.819502] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1618.839065] Lustre: Skipped 1 previous similar message [ 1624.043125] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1624.058604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1624.064049] Lustre: Skipped 1 previous similar message [ 1624.091262] Lustre: Skipped 7 previous similar messages [ 1624.118318] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1624.131967] Lustre: Skipped 1 previous similar message [ 1624.198727] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:854 to 0x2c0000401:897) [ 1624.211680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:853 to 0x280000401:897) [ 1624.727366] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1635.490659] Lustre: DEBUG MARKER: == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 09:05:46 (1783947946) [ 1651.880243] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1671.832881] Lustre: 52967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1697.916110] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1701.604914] Lustre: 54103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1708.855713] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1708.864820] Lustre: Skipped 81 previous similar messages [ 1711.934407] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1712.992398] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1714.015615] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1716.063188] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1716.064906] Lustre: Skipped 1 previous similar message [ 1719.337897] Lustre: Failing over lustre-MDT0000 [ 1719.728245] Lustre: server umount lustre-MDT0000 complete [ 1721.319459] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.342102] Lustre: Skipped 15 previous similar messages [ 1721.353180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1731.046061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1736.736311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:947 to 0x280000401:993) [ 1736.736458] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:946 to 0x2c0000401:961) [ 1737.115741] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1747.448571] Lustre: DEBUG MARKER: == sanity-lfsck test 8: LFSCK state machine ============== 09:07:38 (1783948058) [ 1752.059844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1752.069506] Lustre: Skipped 5 previous similar messages [ 1755.859068] Lustre: server umount lustre-MDT0000 complete [ 1759.947293] LustreError: 41034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948072 with bad export cookie 12563524709305405161 [ 1759.963386] LustreError: 41034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1760.264140] Lustre: server umount lustre-MDT0001 complete [ 1774.147324] Lustre: server umount lustre-OST0000 complete [ 1778.465543] Lustre: 16389:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783948075/real 1783948075] req@ffff8a75fafd3800 x1870603842883456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783948091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1779.970071] Lustre: server umount lustre-OST0001 complete [ 1789.133979] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 1798.249574] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1850.126645] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1861.518543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.825748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1861.870979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1861.976870] Lustre: lustre-MDT0000: new disk, initializing [ 1862.065284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1867.140447] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1878.816917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.934592] Lustre: 59164:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1878.963823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1878.970542] Lustre: Skipped 1 previous similar message [ 1879.046320] Lustre: lustre-MDT0001: new disk, initializing [ 1879.158775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1879.175364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1886.263891] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1892.715341] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1900.072748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1900.508547] Lustre: lustre-OST0000: new disk, initializing [ 1900.516467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1900.524256] Lustre: 60794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1902.022224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1902.025814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1902.132328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1908.478869] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1921.106220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1921.282505] Lustre: lustre-OST0001: new disk, initializing [ 1921.286157] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1921.299270] Lustre: 61665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1922.547894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1922.565372] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1922.695247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1929.415176] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1942.044534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1946.838112] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1961.849108] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1963.079324] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1963.083094] Lustre: Skipped 19 previous similar messages [ 1968.550586] Lustre: *** cfs_fail_loc=1601, val=2*** [ 1968.559659] Lustre: Skipped 6 previous similar messages [ 1991.428508] Lustre: Failing over lustre-MDT0000 [ 1991.650405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1991.654480] LustreError: Skipped 1 previous similar message [ 1991.702296] Lustre: server umount lustre-MDT0000 complete [ 2001.800174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2001.930668] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2001.944342] LustreError: Skipped 3 previous similar messages [ 2002.298612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.316688] Lustre: Skipped 7 previous similar messages [ 2002.376977] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2002.398569] Lustre: Skipped 1 previous similar message [ 2006.623225] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2007.524379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2007.532517] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2007.533654] Lustre: Skipped 7 previous similar messages [ 2007.554141] Lustre: Skipped 1 previous similar message [ 2007.570137] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2007.589199] Lustre: Skipped 1 previous similar message [ 2007.646291] Lustre: *** cfs_fail_loc=160b, val=2*** [ 2007.652419] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:65) [ 2007.660296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:65) [ 2017.043869] Lustre: Failing over lustre-MDT0000 [ 2017.283591] Lustre: server umount lustre-MDT0000 complete [ 2017.764560] LustreError: 62451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.782466] LustreError: 62451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 40 previous similar messages [ 2028.738272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2034.236912] Lustre: *** cfs_fail_loc=160b, val=2*** [ 2034.247932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:97) [ 2034.248097] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:97) [ 2035.033987] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2040.448787] Lustre: Failing over lustre-MDT0000 [ 2040.644280] Lustre: server umount lustre-MDT0000 complete [ 2048.860707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2054.146378] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2054.676328] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:129) [ 2054.682507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:129) [ 2060.075686] Lustre: *** cfs_fail_loc=1602, val=2*** [ 2060.078273] Lustre: Skipped 3 previous similar messages [ 2073.372242] Lustre: DEBUG MARKER: == sanity-lfsck test 9a: LFSCK speed control (1) ========= 09:13:04 (1783948384) [ 2089.745106] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2109.038667] Lustre: 68689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2137.395504] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2141.810502] Lustre: 69825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2150.893953] Lustre: 59169:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 2150.898076] Lustre: 59169:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 2807 previous similar messages [ 2150.900912] Lustre: 59169:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 2150.903912] Lustre: 59169:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 2808 previous similar messages [ 2150.906712] Lustre: 59169:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 2150.909412] Lustre: 59169:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 2808 previous similar messages [ 2150.912400] Lustre: 59169:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 2150.915221] Lustre: 59169:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 2808 previous similar messages [ 2150.918503] Lustre: 59169:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 2150.921418] Lustre: 59169:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 2808 previous similar messages [ 2150.925271] Lustre: 59169:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 2150.928514] Lustre: 59169:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 2808 previous similar messages [ 2261.322868] Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) ========= 09:16:12 (1783948572) [ 2319.850689] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2319.853531] Lustre: Skipped 4 previous similar messages [ 2346.841451] Lustre: *** cfs_fail_loc=160c, val=0*** [ 2346.843402] Lustre: Skipped 10 previous similar messages [ 2387.264903] Lustre: DEBUG MARKER: == sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 09:18:18 (1783948698) [ 2431.371634] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2432.388439] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2432.390896] Lustre: Skipped 36 previous similar messages [ 2434.391278] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2434.393191] Lustre: Skipped 100 previous similar messages [ 2438.391897] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2438.395716] Lustre: Skipped 171 previous similar messages [ 2446.409118] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2446.410896] Lustre: Skipped 376 previous similar messages [ 2462.413670] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2462.416484] Lustre: Skipped 894 previous similar messages [ 2494.418845] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2494.420670] Lustre: Skipped 1607 previous similar messages [ 2503.136626] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2503.143963] Lustre: Skipped 2599 previous similar messages [ 2763.279395] Lustre: DEBUG MARKER: == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 09:24:33 (1783949073) [ 2790.495755] Lustre: 59171:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 320, rollback = 2 [ 2790.506465] Lustre: 59171:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 36831 previous similar messages [ 2790.512473] Lustre: 59171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 1/4/0 [ 2790.523125] Lustre: 59171:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2790.531494] Lustre: 59171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 5/5/0, xattr_set: 7/320/0 [ 2790.543014] Lustre: 59171:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2790.555605] Lustre: 59171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 6/46/0, punch: 0/0/0, quota 1/3/0 [ 2790.563979] Lustre: 59171:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2790.570361] Lustre: 59171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 2/5/1 [ 2790.575430] Lustre: 59171:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2790.582509] Lustre: 59171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 2790.588465] Lustre: 59171:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2909.664466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2909.672312] LustreError: Skipped 2 previous similar messages [ 2909.675033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.675730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2909.681862] Lustre: Skipped 19 previous similar messages [ 2909.697051] Lustre: Skipped 1 previous similar message [ 2913.689612] Lustre: server umount lustre-MDT0000 complete [ 2914.794483] LustreError: 70258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.804749] LustreError: 70258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 2917.335480] LustreError: 69827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949230 with bad export cookie 12563524709305424250 [ 2917.343858] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2917.348185] LustreError: 69827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2917.359199] LustreError: Skipped 2 previous similar messages [ 2917.930285] Lustre: server umount lustre-MDT0001 complete [ 2931.300138] Lustre: server umount lustre-OST0000 complete [ 2945.665209] Lustre: server umount lustre-OST0001 complete [ 2953.510147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 2963.869867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2979.487552] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2984.607347] LustreError: 74732:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 4: rc = -110 [ 3010.335416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.348734] Lustre: Skipped 2 previous similar messages [ 3018.924470] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3023.001621] Lustre: 75315:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 3023.027560] Lustre: *** cfs_fail_loc=160e, val=3*** [ 3026.092414] Lustre: 75315:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 3036.981475] Lustre: DEBUG MARKER: == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 09:29:08 (1783949348) [ 3052.841638] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3065.971743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.462854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3272 to 0x280000401:3297) [ 3071.607821] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3081.396735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.953703] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3090.447514] Lustre: 77982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3110.024549] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3115.524749] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3207 to 0x2c0000401:3233) [ 3118.842778] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3127.656577] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3131.448445] Lustre: 79481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3136.434159] Lustre: *** cfs_fail_loc=160d, val=0*** [ 3137.007476] Lustre: *** cfs_fail_loc=160d, val=0*** [ 3137.009373] Lustre: Skipped 3 previous similar messages [ 3144.555646] Lustre: Failing over lustre-OST0000 [ 3144.661273] Lustre: server umount lustre-OST0000 complete [ 3146.214821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.231457] Lustre: Skipped 2 previous similar messages [ 3156.437325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3156.763359] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3156.779494] Lustre: Skipped 2 previous similar messages [ 3158.403505] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3158.426781] Lustre: Skipped 2 previous similar messages [ 3158.477090] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3158.477360] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3158.477732] Lustre: *** cfs_fail_loc=215, val=0*** [ 3158.486624] Lustre: Skipped 11 previous similar messages [ 3158.511699] Lustre: Skipped 2 previous similar messages [ 3163.617543] Lustre: *** cfs_fail_loc=215, val=0*** [ 3163.638606] Lustre: Skipped 1 previous similar message [ 3165.755168] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3168.738284] Lustre: *** cfs_fail_loc=215, val=0*** [ 3168.745927] Lustre: Skipped 1 previous similar message [ 3170.778942] Lustre: 80884:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 3170.799719] Lustre: 80884:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 3173.862744] Lustre: *** cfs_fail_loc=215, val=0*** [ 3173.867253] Lustre: Skipped 1 previous similar message [ 3173.962740] Lustre: Failing over lustre-OST0000 [ 3174.079739] Lustre: server umount lustre-OST0000 complete [ 3185.065759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3186.630133] Lustre: *** cfs_fail_loc=215, val=0*** [ 3186.631785] Lustre: Skipped 1 previous similar message [ 3192.891599] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3196.897619] Lustre: *** cfs_fail_loc=215, val=0*** [ 3196.899465] Lustre: Skipped 2 previous similar messages [ 3201.002439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3201.016266] Lustre: Skipped 5 previous similar messages [ 3206.254543] Lustre: server umount lustre-MDT0000 complete [ 3210.669967] LustreError: 74738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949523 with bad export cookie 12563524709307008798 [ 3210.677123] LustreError: 74738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3211.164529] Lustre: server umount lustre-MDT0001 complete [ 3227.141995] Lustre: server umount lustre-OST0000 complete [ 3227.619823] Lustre: 16391:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783949524/real 1783949524] req@ffff8a74cba8ce00 x1870603846763904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783949540 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3231.460317] Lustre: 16389:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783949528/real 1783949528] req@ffff8a74cb2b2d80 x1870603846764288/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783949544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3232.141044] Lustre: server umount lustre-OST0001 complete [ 3242.191277] Lustre: DEBUG MARKER: == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 09:32:33 (1783949553) [ 3258.804205] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3272.584533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3279.283971] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3290.000103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.166560] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3299.764334] Lustre: 85271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3306.743132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3314.571859] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3318.463596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3362 to 0x280000401:3393) [ 3324.234974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3329.537041] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3207 to 0x2c0000401:3265) [ 3332.245205] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3340.410334] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3344.719538] Lustre: 87138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3411.662546] Lustre: DEBUG MARKER: == sanity-lfsck test 12b: auto detect Lustre device ====== 09:35:22 (1783949722) [ 3413.725324] Lustre: 87531:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 3413.729534] Lustre: 87531:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 999 previous similar messages [ 3413.734228] Lustre: 87531:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 3413.738282] Lustre: 87531:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 999 previous similar messages [ 3413.741803] Lustre: 87531:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3413.745401] Lustre: 87531:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 999 previous similar messages [ 3413.749834] Lustre: 87531:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3413.757321] Lustre: 87531:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 999 previous similar messages [ 3413.760423] Lustre: 87531:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 3413.763496] Lustre: 87531:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 999 previous similar messages [ 3413.767554] Lustre: 87531:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3413.771346] Lustre: 87531:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 999 previous similar messages [ 3428.512175] Lustre: DEBUG MARKER: == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 09:35:39 (1783949739) [ 3429.843131] Lustre: *** cfs_fail_loc=160f, val=0*** [ 3442.750582] Lustre: DEBUG MARKER: == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 09:35:53 (1783949753) [ 3447.451138] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3447.453247] Lustre: Skipped 3 previous similar messages [ 3503.599231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.604922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3503.619909] Lustre: Skipped 9 previous similar messages [ 3503.642214] Lustre: Skipped 3 previous similar messages [ 3507.282847] Lustre: server umount lustre-MDT0000 complete [ 3512.841372] LustreError: 84109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949825 with bad export cookie 12563524709307017247 [ 3512.853343] LustreError: 84109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3513.269586] Lustre: server umount lustre-MDT0001 complete [ 3528.399676] Lustre: server umount lustre-OST0000 complete [ 3529.185865] Lustre: 16391:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783949826/real 1783949826] req@ffff8a74c1724a80 x1870603847018368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783949842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3533.288094] Lustre: 16389:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783949830/real 1783949830] req@ffff8a74caa1a300 x1870603847018624/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783949846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3534.120544] Lustre: server umount lustre-OST0001 complete [ 3561.521071] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3577.245898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.925975] LustreError: 92572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3577.959494] LustreError: 92572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 50 previous similar messages [ 3583.851460] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3593.890634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3599.846572] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3603.773944] Lustre: 93713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3612.052991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3612.577634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3612.590567] Lustre: Skipped 11 previous similar messages [ 3619.751387] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:161) [ 3621.544415] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3622.829738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3554 to 0x280000401:3585) [ 3633.085964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3638.781866] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:161) [ 3638.790107] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3361) [ 3641.634824] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3649.485657] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3654.009073] Lustre: 95586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3662.013306] Lustre: DEBUG MARKER: == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 09:39:32 (1783949972) [ 3667.438909] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3667.450437] Lustre: Skipped 63 previous similar messages [ 3695.074619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3695.081100] Lustre: Skipped 3 previous similar messages [ 3700.193771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3700.211551] Lustre: Skipped 3 previous similar messages [ 3701.168785] Lustre: server umount lustre-MDT0000 complete [ 3706.691419] LustreError: 92552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783950019 with bad export cookie 12563524709307045653 [ 3706.705242] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3706.716656] LustreError: Skipped 2 previous similar messages [ 3707.317751] Lustre: server umount lustre-MDT0001 complete [ 3722.424765] Lustre: server umount lustre-OST0000 complete [ 3737.219289] Lustre: server umount lustre-OST0001 complete [ 3757.779907] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3770.553792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3777.908913] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3789.522040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3794.458545] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3797.903553] Lustre: 99628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3806.662035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3814.261922] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:193) [ 3815.656700] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3816.313129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3682 to 0x280000401:3713) [ 3827.826684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3833.324512] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3393) [ 3833.351022] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:193) [ 3837.158754] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3847.254380] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3859.548801] Lustre: DEBUG MARKER: == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 09:42:51 (1783950171) [ 3863.341602] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3863.348057] Lustre: Skipped 63 previous similar messages [ 3863.753792] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3882.588203] Lustre: DEBUG MARKER: == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 09:43:12 (1783950192) [ 3886.909464] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3887.035993] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3887.053606] Lustre: Skipped 3 previous similar messages [ 3902.178868] Lustre: DEBUG MARKER: == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 09:43:32 (1783950212) [ 3904.012442] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_15c MDS newer than 2.7.55, LU-6475 [ 3906.629507] Lustre: DEBUG MARKER: == sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 09:43:37 (1783950217) [ 3914.449701] Lustre: *** cfs_fail_loc=1709, val=0*** [ 3914.458834] LustreError: 98496:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200003ab2:0x1:0x0]: rc = -5 [ 3914.471776] LustreError: 98496:0:(mdt_reint.c:2644:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240002340:0x6c:0x0]/s35 failed: rc = -5 [ 3989.472376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3989.484683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.496760] Lustre: Skipped 7 previous similar messages [ 3989.504364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3993.761160] Lustre: server umount lustre-MDT0000 complete [ 4003.828390] LustreError: 103020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783950316 with bad export cookie 12563524709307060381 [ 4003.843613] LustreError: 103020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4004.274873] Lustre: server umount lustre-MDT0001 complete [ 4023.778160] Lustre: 16392:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783950320/real 1783950320] req@ffff8a74caa12a00 x1870603847809920/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783950336 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4025.098695] Lustre: server umount lustre-OST0000 complete [ 4034.015650] Lustre: 16391:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783950331/real 1783950331] req@ffff8a74caa12680 x1870603847811072/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783950347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4034.055720] Lustre: 16391:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4038.217557] Lustre: server umount lustre-OST0001 complete [ 4060.200335] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 4063.362282] Key type lgssc unregistered [ 4063.629829] LNet: 105338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4063.634424] LNetError: 105338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4063.654239] LNet: Removed LNI 192.168.202.138@tcp [ 4064.797447] Key type .llcrypt unregistered [ 4064.798760] Key type ._llcrypt unregistered [ 4091.740436] Key type ._llcrypt registered [ 4091.742907] Key type .llcrypt registered [ 4091.918459] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 4107.270796] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4109.004411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4109.085591] alg: No test for adler32 (adler32-zlib) [ 4110.301565] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 4110.660553] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 4112.343458] Key type lgssc registered [ 4113.691178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4168.217837] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4183.859234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4183.922362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4185.219358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4185.251827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4185.365086] Lustre: lustre-MDT0000: new disk, initializing [ 4185.479934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.510482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4190.709838] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4205.686124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4205.817541] Lustre: 109797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4205.845383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4205.853274] Lustre: Skipped 1 previous similar message [ 4205.962298] Lustre: lustre-MDT0001: new disk, initializing [ 4206.055065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4206.105169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4206.121321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4210.659191] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4216.338283] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4225.823769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4226.155790] Lustre: lustre-OST0000: new disk, initializing [ 4226.162256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4226.174289] Lustre: 111734:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4226.258659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.191093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4231.202493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4231.246435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4231.901640] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4245.090490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4245.229727] Lustre: lustre-OST0001: new disk, initializing [ 4245.233933] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4245.238607] Lustre: 112758:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4245.297761] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4251.870282] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4252.697924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4252.713484] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4252.783344] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4263.463593] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4270.883786] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4276.748033] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 09:49:48 (1783950588) === [ 4282.595273] Lustre: DEBUG MARKER: == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 09:49:54 (1783950594) [ 4282.777929] Lustre: 109803:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 4282.793985] Lustre: 109803:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4282.807358] Lustre: 109803:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4282.810807] Lustre: 109803:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 4282.816648] Lustre: 109803:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 4282.830421] Lustre: 109803:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4283.279959] Lustre: 111024:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 4283.286303] Lustre: 111024:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 21 previous similar messages [ 4283.300871] Lustre: 113342:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 4283.306654] Lustre: 113342:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 4283.310736] Lustre: 113342:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4283.317708] Lustre: 113342:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 4283.322438] Lustre: 113342:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 7/369/0 [ 4283.327760] Lustre: 113342:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 4283.331213] Lustre: 113342:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 4283.337690] Lustre: 113342:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 4283.341899] Lustre: 113342:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4283.346266] Lustre: 113342:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 4284.296571] Lustre: 109803:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 4284.313314] Lustre: 109803:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 248 previous similar messages [ 4284.323525] Lustre: 109803:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 4284.329410] Lustre: 109803:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 245 previous similar messages [ 4284.336641] Lustre: 109803:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4284.343575] Lustre: 109803:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 245 previous similar messages [ 4284.348366] Lustre: 109803:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 7/369/0 [ 4284.353297] Lustre: 109803:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 245 previous similar messages [ 4284.357681] Lustre: 109803:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 4284.362452] Lustre: 109803:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 245 previous similar messages [ 4284.367773] Lustre: 109803:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4284.372620] Lustre: 109803:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 245 previous similar messages [ 4285.378657] Lustre: *** cfs_fail_loc=1613, val=0*** [ 4296.068294] Lustre: DEBUG MARKER: == sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 09:50:07 (1783950607) [ 4297.312211] Lustre: 109804:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 278, rollback = 2 [ 4297.318675] Lustre: 109804:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 38 previous similar messages [ 4297.321666] Lustre: 109804:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 4297.330241] Lustre: 109804:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 4297.334632] Lustre: 109804:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4297.340650] Lustre: 109804:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 4297.345398] Lustre: 109804:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4297.350292] Lustre: 109804:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 4297.355969] Lustre: 109804:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 4297.361673] Lustre: 109804:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 4297.366396] Lustre: 109804:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4297.371475] Lustre: 109804:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 38 previous similar messages [ 4298.489434] Lustre: *** cfs_fail_loc=1614, val=0*** [ 4302.942163] Lustre: 111724:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 275, rollback = 2 [ 4302.953389] Lustre: 111724:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 11 previous similar messages [ 4302.966452] Lustre: 111724:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4302.970181] Lustre: 111724:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4302.978438] Lustre: 111724:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 3/275/0 [ 4302.983392] Lustre: 111724:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4302.987917] Lustre: 111724:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 1/4/0, quota 7/225/0 [ 4302.991221] Lustre: 111724:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4302.995598] Lustre: 111724:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 4302.999694] Lustre: 111724:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4303.004467] Lustre: 111724:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4303.008761] Lustre: 111724:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4309.466694] Lustre: DEBUG MARKER: == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 09:50:20 (1783950620) [ 4311.015697] Lustre: 111722:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 274, rollback = 2 [ 4311.030466] Lustre: 111722:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 19 previous similar messages [ 4311.037617] Lustre: 111722:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4311.038860] Lustre: 114791:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 4311.041965] Lustre: 111722:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 20 previous similar messages [ 4311.042016] Lustre: 111722:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 4311.042020] Lustre: 111722:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 19 previous similar messages [ 4311.042054] Lustre: 111722:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 4311.042057] Lustre: 111722:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 19 previous similar messages [ 4311.042064] Lustre: 111722:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4311.042067] Lustre: 111722:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 19 previous similar messages [ 4311.106834] Lustre: 114791:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 20 previous similar messages [ 4312.156288] Lustre: *** cfs_fail_loc=1615, val=0*** [ 4312.158218] Lustre: Skipped 1 previous similar message [ 4313.160714] Lustre: *** cfs_fail_loc=1615, val=0*** [ 4313.164658] Lustre: Skipped 3 previous similar messages [ 4332.574535] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_18b skipping excluded test 18b [ 4334.421797] Lustre: DEBUG MARKER: == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 09:50:45 (1783950645) [ 4334.972579] Lustre: 109804:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 4334.979659] Lustre: 109804:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 4334.984966] Lustre: 109804:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 4334.994613] Lustre: 109804:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 4335.003616] Lustre: 109804:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4335.009356] Lustre: 109804:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 4335.015648] Lustre: 109804:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4335.023222] Lustre: 109804:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 4335.029832] Lustre: 109804:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 4335.035800] Lustre: 109804:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 4335.039214] Lustre: 109804:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4335.043206] Lustre: 109804:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 4337.703164] Lustre: *** cfs_fail_loc=1617, val=0*** [ 4337.776927] Lustre: *** cfs_fail_loc=1617, val=0*** [ 4340.170756] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4340.173836] Lustre: Skipped 1 previous similar message [ 4359.934384] Lustre: DEBUG MARKER: == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 09:51:11 (1783950671) [ 4362.093888] Lustre: *** cfs_fail_loc=1618, val=0*** [ 4362.095322] Lustre: Skipped 5 previous similar messages [ 4400.611951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4400.631413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4400.643055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4401.138774] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.145712] Lustre: Skipped 2 previous similar messages [ 4401.148861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4401.153636] Lustre: Skipped 3 previous similar messages [ 4402.504076] Lustre: server umount lustre-MDT0000 complete [ 4405.826815] LustreError: 109789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783950718 with bad export cookie 4292482075790282638 [ 4405.830862] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4405.832929] LustreError: 109789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4406.030605] Lustre: server umount lustre-MDT0001 complete [ 4420.367853] Lustre: server umount lustre-OST0000 complete [ 4422.623248] Lustre: 106955:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783950719/real 1783950719] req@ffff8a74cce87100 x1870607598954240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783950735 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4422.667935] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4424.148626] Lustre: server umount lustre-OST0001 complete [ 4442.926940] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4456.442236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4457.200079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4462.559638] LustreError: 118466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4462.593808] LustreError: 118466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4462.836421] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4467.682666] LustreError: 118466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4471.969428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.367888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4477.911828] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4481.484088] Lustre: 119609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4488.355987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.670831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.721800] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4497.896872] LustreError: 119963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4497.941470] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:33) [ 4502.005358] LustreError: 119964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4502.043828] LustreError: 119964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4502.051862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:116 to 0x280000401:161) [ 4504.828632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4510.191818] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:33) [ 4510.197863] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 4511.911964] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4520.060679] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4523.678574] Lustre: 121477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4537.728718] Lustre: DEBUG MARKER: == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 09:54:09 (1783950849) [ 4538.021739] Lustre: 118462:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 4538.030893] Lustre: 118462:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 46 previous similar messages [ 4538.036891] Lustre: 118462:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 4538.046271] Lustre: 118462:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4538.053620] Lustre: 118462:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4538.059288] Lustre: 118462:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4538.065412] Lustre: 118462:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4538.071144] Lustre: 118462:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4538.076110] Lustre: 118462:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 4538.083620] Lustre: 118462:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4538.089922] Lustre: 118462:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4538.095235] Lustre: 118462:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 46 previous similar messages [ 4540.058966] Lustre: *** cfs_fail_loc=1618, val=0*** [ 4540.061926] Lustre: Skipped 3 previous similar messages [ 4574.688600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4574.706833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4574.736929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4576.201571] Lustre: server umount lustre-MDT0000 complete [ 4576.736643] LustreError: 118467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4576.747073] LustreError: 118467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4580.201670] LustreError: 118446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783950893 with bad export cookie 4292482075790297877 [ 4580.202752] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4580.225453] LustreError: 118446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4580.539069] Lustre: server umount lustre-MDT0001 complete [ 4594.582996] Lustre: server umount lustre-OST0000 complete [ 4597.608768] Lustre: 106955:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783950894/real 1783950894] req@ffff8a75febb9f80 x1870607599031168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783950910 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4597.643164] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4597.660601] Lustre: Skipped 3 previous similar messages [ 4598.425749] Lustre: server umount lustre-OST0001 complete [ 4615.341412] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4626.957284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.405446] LustreError: 124043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4627.492451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4627.499766] Lustre: Skipped 1 previous similar message [ 4632.321821] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4641.235909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4646.671586] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4649.855766] Lustre: 125182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4656.956784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4663.184989] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4667.558058] LustreError: 125536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4667.565520] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:65) [ 4667.598060] LustreError: 125536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4672.379915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4673.658179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4673.669719] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 4673.689884] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:65) [ 4678.878100] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4685.868023] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4689.691720] Lustre: 127052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4694.560514] Lustre: 125841:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 252 < left 260, rollback = 2 [ 4694.567522] Lustre: 125841:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 21 previous similar messages [ 4694.573658] Lustre: 125841:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4694.584184] Lustre: 125841:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4694.594426] Lustre: 125841:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/260/0 [ 4694.603292] Lustre: 125841:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4694.609370] Lustre: 125841:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 0/0/0, quota 4/150/2 [ 4694.615302] Lustre: 125841:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4694.625247] Lustre: 125841:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 1/17/2, delete: 0/0/0 [ 4694.634084] Lustre: 125841:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4694.640242] Lustre: 125841:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4694.645793] Lustre: 125841:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4694.720586] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4718.426104] Lustre: DEBUG MARKER: == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 09:57:09 (1783951029) [ 4721.058553] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4721.060861] Lustre: Skipped 3 previous similar messages [ 4727.981314] Lustre: *** cfs_fail_loc=161c, val=0*** [ 4745.564559] Lustre: DEBUG MARKER: == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) ========================================================== 09:57:36 (1783951056) [ 4747.704099] Lustre: *** cfs_fail_loc=162e, val=0*** [ 4761.860367] Lustre: DEBUG MARKER: == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 09:57:53 (1783951073) [ 4765.571316] Lustre: *** cfs_fail_loc=162f, val=0*** [ 4765.577084] Lustre: Skipped 9 previous similar messages [ 4779.899406] Lustre: DEBUG MARKER: == sanity-lfsck test 19a: OST-object inconsistency self detect ========================================================== 09:58:11 (1783951091) [ 4791.968647] Lustre: DEBUG MARKER: == sanity-lfsck test 19b: OST-object inconsistency self repair ========================================================== 09:58:23 (1783951103) [ 4795.621112] Lustre: *** cfs_fail_loc=1611, val=0*** [ 4795.674691] Lustre: *** cfs_fail_loc=1611, val=0*** [ 4795.680856] Lustre: Skipped 3 previous similar messages [ 4801.357366] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.38@tcp inode [0x2000013a1:0x1f:0x0] object 0x280000401:203 extent [0-4095], client returned csum 0 (type 4), server csum 703fb3ad (type 4) [ 4802.509298] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.38@tcp inode [0x2000013a1:0x20:0x0] object 0x280000401:204 extent [0-4095], client returned csum 0 (type 4), server csum 7cbb935c (type 4) [ 4810.821030] Lustre: DEBUG MARKER: == sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================================================== 09:58:42 (1783951122) [ 4828.023898] Lustre: 131581:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 263, rollback = 2 [ 4828.029741] Lustre: 131581:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 125 previous similar messages [ 4828.034376] Lustre: 131581:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4828.039539] Lustre: 131581:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4828.045872] Lustre: 131581:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 2/263/0 [ 4828.050457] Lustre: 131581:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4828.055988] Lustre: 131581:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/2 [ 4828.068691] Lustre: 131581:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4828.073083] Lustre: 131581:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 4828.079230] Lustre: 131581:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4828.084650] Lustre: 131581:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4828.089339] Lustre: 131581:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 125 previous similar messages [ 4842.783526] Lustre: DEBUG MARKER: == sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case ========================================================== 09:59:13 (1783951153) [ 4849.156259] Lustre: DEBUG MARKER: == sanity-lfsck test 21: run all LFSCK components by default ========================================================== 09:59:20 (1783951160) [ 4860.925766] Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) ========================================================== 09:59:31 (1783951171) [ 4863.246725] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4863.265400] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4863.269812] Lustre: Skipped 1 previous similar message [ 4873.870284] Lustre: DEBUG MARKER: == sanity-lfsck test 22b: LFSCK can repair unmatched pairs (2) ========================================================== 09:59:45 (1783951185) [ 4875.386903] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4875.389149] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4885.704085] Lustre: DEBUG MARKER: == sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) ========================================================== 09:59:57 (1783951197) [ 4887.618785] Lustre: *** cfs_fail_loc=1620, val=0*** [ 4901.908895] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_23b skipping excluded test 23b [ 4904.624986] Lustre: DEBUG MARKER: == sanity-lfsck test 23c: LFSCK can repair dangling name entry (3) ========================================================== 10:00:14 (1783951214) [ 4913.097670] Lustre: *** cfs_fail_loc=1621, val=127*** [ 4913.101740] Lustre: Skipped 1 previous similar message [ 4916.900575] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4938.127866] Lustre: DEBUG MARKER: == sanity-lfsck test 23d: LFSCK can repair a dangling name entry to a remote object ========================================================== 10:00:49 (1783951249) [ 4939.878513] Lustre: Failing over lustre-MDT0000 [ 4940.093128] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 4940.268746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.300650] LustreError: 124043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4940.443993] Lustre: server umount lustre-MDT0000 complete [ 4949.765507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.017820] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4950.446358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4950.451672] Lustre: Skipped 3 previous similar messages [ 4950.479251] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4950.812326] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4954.405486] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4955.626277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4955.654558] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4955.704886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 4955.706584] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:130 to 0x2c0000401:161) [ 4956.183178] LustreError: 124038:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0000: [0x2000013a3:0x82:0x0] doesn't exist!: rc = -14 [ 4966.691140] Lustre: DEBUG MARKER: == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ========================================================== 10:01:17 (1783951277) [ 4968.739919] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4968.890700] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4968.895514] Lustre: Skipped 1 previous similar message [ 4981.878772] Lustre: DEBUG MARKER: == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ========================================================== 10:01:32 (1783951292) [ 4983.345841] Lustre: *** cfs_fail_loc=1623, val=0*** [ 4994.082236] Lustre: DEBUG MARKER: == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace ========================================================== 10:01:45 (1783951305) [ 4995.896159] Lustre: *** cfs_fail_loc=1624, val=0*** [ 5005.766360] Lustre: DEBUG MARKER: == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========================================================== 10:01:57 (1783951317) [ 5019.024234] Lustre: DEBUG MARKER: == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan ========================================================== 10:02:10 (1783951330) [ 5020.882768] Lustre: *** cfs_fail_loc=1624, val=0*** [ 5020.890312] Lustre: Skipped 1 previous similar message [ 5033.445575] Lustre: DEBUG MARKER: == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ========================================================== 10:02:24 (1783951344) [ 5047.254366] Lustre: DEBUG MARKER: == sanity-lfsck test 28: Skip the failed MDT(s) when handle orphan MDT-objects ========================================================== 10:02:38 (1783951358) [ 5054.399401] Lustre: *** cfs_fail_loc=161c, val=0*** [ 5054.402848] Lustre: Skipped 1 previous similar message [ 5071.421462] Lustre: DEBUG MARKER: == sanity-lfsck test 29b: LFSCK can repair bad nlink count (2) ========================================================== 10:03:02 (1783951382) [ 5073.405137] Lustre: *** cfs_fail_loc=1626, val=0*** [ 5073.410974] Lustre: Skipped 4 previous similar messages [ 5085.027626] Lustre: DEBUG MARKER: == sanity-lfsck test 29c: verify linkEA size limitation == 10:03:15 (1783951395) [ 5085.413030] Lustre: 124039:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 320, rollback = 2 [ 5085.435148] Lustre: 124039:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 508 previous similar messages [ 5085.451489] Lustre: 124039:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 1/4/0 [ 5085.463406] Lustre: 124039:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 508 previous similar messages [ 5085.481639] Lustre: 124039:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 5/5/1, xattr_set: 7/320/0 [ 5085.493913] Lustre: 124039:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 508 previous similar messages [ 5085.503143] Lustre: 124039:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 6/46/0, punch: 0/0/0, quota 1/3/0 [ 5085.530019] Lustre: 124039:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 508 previous similar messages [ 5085.547184] Lustre: 124039:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 2/5/1 [ 5085.555197] Lustre: 124039:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 508 previous similar messages [ 5085.562039] Lustre: 124039:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1 [ 5085.566628] Lustre: 124039:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 508 previous similar messages [ 5116.993782] Lustre: DEBUG MARKER: == sanity-lfsck test 29d: accessing non-existing inode shouldn't turn fs read-only (ldiskfs) ========================================================== 10:03:47 (1783951427) [ 5118.937173] LustreError: 124038:0:(osd_handler.c:284:osd_idc_find_or_init()) lustre-MDT0000: cannot lookup FID [0x2000013a3:0x9e:0x0]: rc = -2 [ 5126.055136] Lustre: DEBUG MARKER: == sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 10:03:56 (1783951436) [ 5150.741650] Lustre: Failing over lustre-MDT0000 [ 5151.054765] Lustre: server umount lustre-MDT0000 complete [ 5155.297456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5155.299342] LustreError: 124334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.299911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5155.318529] Lustre: Skipped 5 previous similar messages [ 5155.350969] LustreError: 124334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5164.622452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5164.976674] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5165.465443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5165.537775] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5170.660129] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5170.685621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5170.693165] Lustre: Skipped 3 previous similar messages [ 5170.724448] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5170.783423] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:167 to 0x2c0000401:193) [ 5170.795931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:294 to 0x280000401:321) [ 5171.914644] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5187.337675] Lustre: DEBUG MARKER: == sanity-lfsck test 31a: The LFSCK can find/repair the name entry with bad name hash (1) ========================================================== 10:04:57 (1783951497) [ 5202.668696] Lustre: DEBUG MARKER: == sanity-lfsck test 31b: The LFSCK can find/repair the name entry with bad name hash (2) ========================================================== 10:05:13 (1783951513) [ 5217.295886] Lustre: DEBUG MARKER: == sanity-lfsck test 31c: Re-generate the lost master LMV EA for striped directory ========================================================== 10:05:27 (1783951527) [ 5219.429080] Lustre: *** cfs_fail_loc=1629, val=0*** [ 5219.432620] Lustre: Skipped 7 previous similar messages [ 5235.309162] Lustre: DEBUG MARKER: == sanity-lfsck test 31d: Set broken striped directory (modified after broken) as read-only ========================================================== 10:05:46 (1783951546) [ 5243.908551] Lustre: Failing over lustre-MDT0000 [ 5244.414327] Lustre: server umount lustre-MDT0000 complete [ 5247.456929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5247.460848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5247.473833] Lustre: Skipped 4 previous similar messages [ 5254.279049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5254.402918] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5254.683974] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5259.749928] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5259.761278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5259.785086] Lustre: Skipped 3 previous similar messages [ 5259.808438] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5259.863823] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:167 to 0x2c0000401:225) [ 5259.863896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:294 to 0x280000401:353) [ 5260.297860] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5271.928828] Lustre: Failing over lustre-MDT0000 [ 5272.156583] Lustre: server umount lustre-MDT0000 complete [ 5275.119564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5281.589137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5281.823455] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5282.305561] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5283.596533] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5287.443897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5287.448614] Lustre: Skipped 3 previous similar messages [ 5287.480579] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5287.553517] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:227 to 0x2c0000401:257) [ 5287.557177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:294 to 0x280000401:385) [ 5288.456361] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5298.433365] Lustre: DEBUG MARKER: == sanity-lfsck test 31e: Re-generate the lost slave LMV EA for striped directory (1) ========================================================== 10:06:49 (1783951609) [ 5310.866835] Lustre: DEBUG MARKER: == sanity-lfsck test 31f: Re-generate the lost slave LMV EA for striped directory (2) ========================================================== 10:07:02 (1783951622) [ 5324.370303] Lustre: DEBUG MARKER: == sanity-lfsck test 31g: Repair the corrupted slave LMV EA ========================================================== 10:07:15 (1783951635) [ 5362.280527] Lustre: DEBUG MARKER: == sanity-lfsck test 31h: Repair the corrupted shard's name entry ========================================================== 10:07:53 (1783951673) [ 5363.350044] Lustre: *** cfs_fail_loc=162c, val=0*** [ 5363.353841] Lustre: Skipped 13 previous similar messages [ 5376.309408] Lustre: DEBUG MARKER: == sanity-lfsck test 32a: stop LFSCK when some OST failed ========================================================== 10:08:07 (1783951687) [ 5385.696508] LustreError: 148429:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5388.350215] Lustre: Failing over lustre-OST0000 [ 5388.452707] Lustre: server umount lustre-OST0000 complete [ 5388.768470] LustreError: 148429:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 5388.796179] LustreError: 148429:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5389.796498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.801663] LustreError: 125534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5389.810348] Lustre: Skipped 5 previous similar messages [ 5389.810829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5389.849023] LustreError: 125534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 5391.900958] LustreError: 148429:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 5391.918481] LustreError: 148429:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5392.257048] LustreError: 148429:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout interrupted [ 5403.481831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5403.649585] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5405.089807] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5405.166692] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5405.170377] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5405.221352] Lustre: Skipped 3 previous similar messages [ 5410.784482] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5420.988995] Lustre: DEBUG MARKER: == sanity-lfsck test 32b: stop LFSCK when some MDT failed ========================================================== 10:08:52 (1783951732) [ 5435.679166] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5457.955047] Lustre: 151235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5483.125641] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5487.507652] Lustre: 152370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5500.251615] LustreError: 152510:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5503.272522] Lustre: Failing over lustre-MDT0001 [ 5503.287085] LustreError: 152510:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 5503.329960] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 5503.340055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5503.574396] Lustre: server umount lustre-MDT0001 complete [ 5506.426294] LustreError: 152509:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout interrupted [ 5518.759247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5519.086934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5519.093390] Lustre: Skipped 3 previous similar messages [ 5519.136561] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5524.458260] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5524.459058] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5524.466762] Lustre: Skipped 1 previous similar message [ 5524.497358] Lustre: lustre-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5524.600723] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:97) [ 5524.613524] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:97) [ 5524.850805] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5536.022290] Lustre: DEBUG MARKER: == sanity-lfsck test 33: check LFSCK paramters =========== 10:10:46 (1783951846) [ 5554.478867] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5574.344282] Lustre: 155228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5601.541185] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5605.520366] Lustre: 156365:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5607.465204] Lustre: 127818:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 5607.471786] Lustre: 127818:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1208 previous similar messages [ 5607.474974] Lustre: 127818:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 5607.493940] Lustre: 127818:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1208 previous similar messages [ 5607.503295] Lustre: 127818:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5607.509056] Lustre: 127818:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1208 previous similar messages [ 5607.514595] Lustre: 127818:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 5607.520592] Lustre: 127818:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1208 previous similar messages [ 5607.526250] Lustre: 127818:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 5607.532135] Lustre: 127818:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1208 previous similar messages [ 5607.537551] Lustre: 127818:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5607.547330] Lustre: 127818:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1208 previous similar messages [ 5627.725080] Lustre: DEBUG MARKER: == sanity-lfsck test 34: LFSCK can rebuild the lost agent object ========================================================== 10:12:19 (1783951939) [ 5630.129649] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_34 Only valid for ZFS backend [ 5631.876960] Lustre: DEBUG MARKER: == sanity-lfsck test 35: LFSCK can rebuild the lost agent entry ========================================================== 10:12:23 (1783951943) [ 5639.257803] Lustre: *** cfs_fail_loc=1631, val=0*** [ 5652.450733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5652.452504] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.453265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5652.478815] Lustre: Skipped 5 previous similar messages [ 5666.271412] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5666.733858] Lustre: server umount lustre-MDT0000 complete [ 5667.812732] LustreError: 127818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.850095] LustreError: 127818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 5671.127611] LustreError: 136492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783951984 with bad export cookie 4292482075790370691 [ 5671.140795] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5671.157418] LustreError: 136492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5671.746636] Lustre: server umount lustre-MDT0001 complete [ 5686.265687] Lustre: server umount lustre-OST0000 complete [ 5688.033317] Lustre: 106953:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783951985/real 1783951985] req@ffff8a75cf884000 x1870607600430720/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783952001 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5690.722128] Lustre: server umount lustre-OST0001 complete [ 5709.737298] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5722.754125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5728.703247] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5738.463763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5744.747257] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5747.961377] Lustre: 160263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5755.426218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5763.056110] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:129) [ 5763.265626] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5768.178566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:540 to 0x280000401:577) [ 5774.099877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5779.460073] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:129) [ 5779.540551] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:412 to 0x2c0000401:449) [ 5781.413262] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5790.413817] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5795.268385] Lustre: 162134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5805.328207] Lustre: DEBUG MARKER: == sanity-lfsck test 36a: rebuild LOV EA for mirrored file (1) ========================================================== 10:15:16 (1783952116) [ 5807.045805] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36a needs >= 3 OSTs [ 5809.440777] Lustre: DEBUG MARKER: == sanity-lfsck test 36b: rebuild LOV EA for mirrored file (2) ========================================================== 10:15:20 (1783952120) [ 5811.035413] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36b needs >= 3 OSTs [ 5812.777200] Lustre: DEBUG MARKER: == sanity-lfsck test 36c: rebuild LOV EA for mirrored file (3) ========================================================== 10:15:24 (1783952124) [ 5814.331739] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36c needs >= 3 OSTs [ 5816.302775] Lustre: DEBUG MARKER: == sanity-lfsck test 37: LFSCK must skip a ORPHAN ======== 10:15:27 (1783952127) [ 5828.322784] Lustre: DEBUG MARKER: == sanity-lfsck test 38: LFSCK does not break foreign file and reverse is also true ========================================================== 10:15:39 (1783952139) [ 5842.388140] Lustre: DEBUG MARKER: == sanity-lfsck test 39: LFSCK does not break foreign dir and reverse is also true ========================================================== 10:15:53 (1783952153) [ 5859.510639] Lustre: DEBUG MARKER: == sanity-lfsck test 40a: LFSCK correctly fixes lmm_oi in composite layout ========================================================== 10:16:09 (1783952169) [ 5877.049657] Lustre: DEBUG MARKER: == sanity-lfsck test 41: SEL support in LFSCK ============ 10:16:27 (1783952187) [ 5903.502582] Lustre: DEBUG MARKER: == sanity-lfsck test 42: LFSCK repairs inconsistent MDT-object/OST-object encryption flags ========================================================== 10:16:54 (1783952214) [ 5937.937827] Lustre: *** cfs_fail_loc=1632, val=0*** [ 5955.884657] Lustre: DEBUG MARKER: == sanity-lfsck test 43: LFSCK does not loop endlessly on iget failure in scanning-phase1 ========================================================== 10:17:46 (1783952266) [ 5959.045209] Lustre: Failing over lustre-MDT0001 [ 5959.140556] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5959.150894] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5959.160433] Lustre: Skipped 2 previous similar messages [ 5959.165284] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5959.171344] Lustre: Skipped 11 previous similar messages [ 5959.326150] Lustre: server umount lustre-MDT0001 complete [ 5967.352525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5967.830358] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5967.830632] Lustre: lustre-MDT0001: Aborting client recovery [ 5967.842462] LustreError: 165922:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0001: Aborting recovery [ 5967.854744] Lustre: 165946:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5967.859186] Lustre: 165946:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 71ea6a5c-561f-4eb8-a8d2-543f542832b5@ [ 5967.874890] Lustre: lustre-MDT0001: disconnecting 2 stale clients [ 5967.894387] Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] [ 5967.927293] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] [ 5967.978744] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:131 to 0x2c0000400:161) [ 5967.992477] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:161) [ 5972.969807] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5972.985848] Lustre: Skipped 2 previous similar messages [ 5973.009850] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5973.509245] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5977.330033] Lustre: *** cfs_fail_loc=1e2, val=483*** [ 5983.148630] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid [ 5983.544938] Lustre: DEBUG MARKER: osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid in FULL state after 0 sec [ 5990.633989] Lustre: DEBUG MARKER: == sanity-lfsck test 44: umount while lfsck is stopping == 10:18:21 (1783952301) [ 6001.391585] Lustre: *** cfs_fail_loc=1600, val=3*** [ 6004.292946] Lustre: Failing over lustre-MDT0000 [ 6004.536155] Lustre: server umount lustre-MDT0000 complete [ 6014.254362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.348575] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6014.627945] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6014.638344] Lustre: Skipped 2 previous similar messages [ 6017.804087] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6019.524942] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6020.097848] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6020.109893] Lustre: Skipped 2 previous similar messages [ 6020.145424] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6020.221490] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 6020.228989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:621 to 0x280000401:641) [ 6030.666318] Lustre: DEBUG MARKER: == sanity-lfsck test 45: LFSCK should fix UID/GID/PROJID of OST object ========================================================== 10:19:01 (1783952341) [ 6063.261858] Lustre: Failing over lustre-OST0000 [ 6063.402556] Lustre: server umount lustre-OST0000 complete [ 6071.384134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6091.066321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6091.546458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6091.562825] Lustre: Skipped 6 previous similar messages [ 6091.580869] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6092.717346] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6093.426292] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6093.427709] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6093.446786] Lustre: Skipped 3 previous similar messages [ 6099.520184] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6110.961564] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6111.380177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6116.491946] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6116.819840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6125.294664] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff96e91842e000.ost_server_uuid 50 [ 6182.723879] Lustre: DEBUG MARKER: rpc test_45: @@@@@@ FAIL: can't put import for osc.lustre-OST0000-osc-ffff96e91842e000.ost_server_uuid into FULL state after 50 sec, have IDLE [ 6192.308981] Lustre: DEBUG MARKER: sanity-lfsck test_45: @@@@@@ FAIL: client: import is not in FULL state after 50 [ 6204.386612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6204.395632] Lustre: Skipped 3 previous similar messages [ 6207.118800] Lustre: server umount lustre-MDT0000 complete [ 6209.505060] LustreError: 159116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6209.522937] LustreError: 159116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 43 previous similar messages [ 6215.697859] LustreError: 159865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783952528 with bad export cookie 4292482075790453473 [ 6215.702562] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6215.715165] LustreError: 159865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6216.432844] Lustre: server umount lustre-MDT0001 complete [ 6235.104528] Lustre: 106955:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952532/real 1783952532] req@ffff8a75cf885880 x1870607600825216/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783952548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6236.127187] Lustre: 106952:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952533/real 1783952533] req@ffff8a75ef75f480 x1870607600825600/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783952549 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6236.309968] Lustre: server umount lustre-OST0000 complete [ 6240.228945] Lustre: 106954:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952537/real 1783952537] req@ffff8a75cf887800 x1870607600825856/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783952553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6245.263527] Lustre: server umount lustre-OST0001 complete [ 6267.126645] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 6270.893519] Key type lgssc unregistered [ 6271.284760] LNet: 173880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6271.293382] LNetError: 173880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6271.325570] LNet: Removed LNI 192.168.202.138@tcp [ 6272.224167] Key type .llcrypt unregistered [ 6272.226948] Key type ._llcrypt unregistered [ 6299.152954] Key type ._llcrypt registered [ 6299.156145] Key type .llcrypt registered [ 6299.214930] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 6315.232935] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6316.251963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6316.325932] alg: No test for adler32 (adler32-zlib) [ 6317.342850] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 6317.597168] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 6319.303184] Key type lgssc registered [ 6320.768420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6378.137731] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6394.572911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6394.635204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6396.297363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6396.385276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6396.577576] Lustre: lustre-MDT0000: new disk, initializing [ 6396.738433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6396.756628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6402.998926] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6417.845635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6417.947230] Lustre: 178338:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6417.987837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6417.994876] Lustre: Skipped 1 previous similar message [ 6418.098963] Lustre: lustre-MDT0001: new disk, initializing [ 6418.181253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6418.216767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6418.223639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6423.614919] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6428.825035] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6440.479538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6440.804651] Lustre: lustre-OST0000: new disk, initializing [ 6440.809400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6440.821142] Lustre: 180276:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6440.905501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6442.032798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6442.046216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6442.132781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6448.980801] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6464.154716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6464.280883] Lustre: lustre-OST0001: new disk, initializing [ 6464.286231] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6464.292940] Lustre: 181300:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6464.359380] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6471.607429] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6472.245715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6472.276254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6472.404278] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6482.944402] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6496.404928] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6504.116697] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 10:26:54 (1783952814) === [ 6506.765588] Lustre: DEBUG MARKER: == sanity-lfsck test complete, duration 6151 sec ========= 10:26:57 (1783952817) [ 6509.360220] Lustre: DEBUG MARKER: === sanity-lfsck: start cleanup 10:26:59 (1783952819) === [ 6515.101222] Lustre: DEBUG MARKER: === sanity-lfsck: finish cleanup 10:27:05 (1783952825) === [ 6523.873893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6523.878151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6523.902614] Lustre: Skipped 1 previous similar message [ 6523.923635] Lustre: Skipped 3 previous similar messages [ 6524.376136] Lustre: server umount lustre-MDT0000 complete [ 6533.499930] LustreError: 181310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783952846 with bad export cookie 3577808993246830326 [ 6533.509536] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6533.513388] LustreError: 181310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6533.832117] Lustre: server umount lustre-MDT0001 complete [ 6550.369398] Lustre: 175501:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952847/real 1783952847] req@ffff8a74c5154700 x1870609913045632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783952863 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6550.438737] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6550.462146] Lustre: Skipped 2 previous similar messages [ 6554.591182] Lustre: 175499:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952851/real 1783952851] req@ffff8a75de35b100 x1870609913045888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783952867 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6554.774961] Lustre: server umount lustre-OST0000 complete [ 6555.615390] Lustre: 175500:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952852/real 1783952852] req@ffff8a75fdb58700 x1870609913046144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783952868 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6559.714253] Lustre: 175498:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952856/real 1783952856] req@ffff8a75fdb58a80 x1870609913046528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783952872 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6563.490593] Lustre: server umount lustre-OST0001 complete [ 6580.906773] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 6584.002846] Key type lgssc unregistered [ 6584.273970] LNet: 184777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6584.281892] LNetError: 184777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6584.297383] LNet: Removed LNI 192.168.202.138@tcp [ 6585.343146] Key type .llcrypt unregistered [ 6585.351054] Key type ._llcrypt unregistered