[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479839068 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.032 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003160] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007937] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4800.06 BogoMIPS (lpj=2400032) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010229] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.013028] SELinux: Initializing. [ 0.014049] *** VALIDATE selinux *** [ 0.022552] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027641] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030125] *** VALIDATE tmpfs *** [ 0.032195] *** VALIDATE proc *** [ 0.033253] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035272] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037029] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039027] Spectre V2 : User space: Vulnerable [ 0.040000] Speculative Store Bypass: Vulnerable [ 0.041781] debug: unmapping init [mem 0xffffffff8c659000-0xffffffff8c660fff] [ 0.044150] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045736] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050026] ... max period: 00007fffffffffff [ 0.051010] ... fixed-purpose events: 3 [ 0.052008] ... event mask: 000000070000000f [ 0.053285] rcu: Hierarchical SRCU implementation. [ 0.055636] smp: Bringing up secondary CPUs ... [ 0.056534] x86: Booting SMP configuration: [ 0.057030] .... node #0, CPUs: #1 #2 #3 [ 0.061077] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064012] smpboot: Total of 4 processors activated (19200.25 BogoMIPS) [ 0.146621] node 0 deferred pages initialised in 81ms [ 0.151110] devtmpfs: initialized [ 0.154293] x86/mm: Memory block size: 128MB [ 0.157758] gcov: version magic: 0x41383552 [ 0.160838] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.166152] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.169457] pinctrl core: initialized pinctrl subsystem [ 0.172244] [ 0.173010] ************************************************************* [ 0.176024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179022] ** ** [ 0.182018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.185024] ** ** [ 0.188021] ** This means that this kernel is built to expose internal ** [ 0.191022] ** IOMMU data structures, which may compromise security on ** [ 0.193020] ** your system. ** [ 0.196022] ** ** [ 0.198020] ** If you see this message and you are not debugging the ** [ 0.201022] ** kernel, report this immediately to your vendor! ** [ 0.203019] ** ** [ 0.206023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.209019] ************************************************************* [ 0.211776] NET: Registered protocol family 16 [ 0.214525] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.217085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.220103] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.225047] cpuidle: using governor menu [ 0.227512] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.230437] PCI: Using configuration type 1 for base access [ 0.233118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.244089] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.246061] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.250161] cryptd: max_cpu_qlen set to 1000 [ 0.253303] ACPI: Added _OSI(Module Device) [ 0.255013] ACPI: Added _OSI(Processor Device) [ 0.256009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.257013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.262000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.268563] ACPI: Interpreter enabled [ 0.270044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.271013] ACPI: Using IOAPIC for interrupt routing [ 0.274113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.275221] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.285380] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.287041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.290049] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.293093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.298271] acpiphp: Slot [2] registered [ 0.300124] acpiphp: Slot [5] registered [ 0.302126] acpiphp: Slot [6] registered [ 0.303120] acpiphp: Slot [7] registered [ 0.305107] acpiphp: Slot [8] registered [ 0.306136] acpiphp: Slot [9] registered [ 0.308125] acpiphp: Slot [10] registered [ 0.310131] acpiphp: Slot [3] registered [ 0.311107] acpiphp: Slot [4] registered [ 0.313248] acpiphp: Slot [11] registered [ 0.315105] acpiphp: Slot [12] registered [ 0.317136] acpiphp: Slot [13] registered [ 0.319114] acpiphp: Slot [14] registered [ 0.320119] acpiphp: Slot [15] registered [ 0.322100] acpiphp: Slot [16] registered [ 0.323193] acpiphp: Slot [17] registered [ 0.325178] acpiphp: Slot [18] registered [ 0.327132] acpiphp: Slot [19] registered [ 0.328132] acpiphp: Slot [20] registered [ 0.330146] acpiphp: Slot [21] registered [ 0.332140] acpiphp: Slot [22] registered [ 0.333169] acpiphp: Slot [23] registered [ 0.334122] acpiphp: Slot [24] registered [ 0.336137] acpiphp: Slot [25] registered [ 0.338188] acpiphp: Slot [26] registered [ 0.340177] acpiphp: Slot [27] registered [ 0.342160] acpiphp: Slot [28] registered [ 0.344246] acpiphp: Slot [29] registered [ 0.346187] acpiphp: Slot [30] registered [ 0.347186] acpiphp: Slot [31] registered [ 0.350163] PCI host bridge to bus 0000:00 [ 0.352039] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.355056] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.357046] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.361029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.364025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.368026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.370163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.374098] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.378525] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.389878] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.396014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.400021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.403028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.406020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.409684] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.414989] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.420063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.424971] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.430023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.444017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.449000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.455892] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.466097] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.476015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.510017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.525772] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.543015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.551035] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.584016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.595000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.605014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.616017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.641020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.651109] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.660014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.669015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.696021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.710603] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.718020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.725015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.744016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.758038] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.768014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.775014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.792014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.804922] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.807404] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.810404] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.813362] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.816240] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.820653] iommu: Default domain type: Passthrough [ 0.822355] SCSI subsystem initialized [ 0.824135] ACPI: bus type USB registered [ 0.825118] usbcore: registered new interface driver usbfs [ 0.828102] usbcore: registered new interface driver hub [ 0.829055] usbcore: registered new device driver usb [ 0.831118] pps_core: LinuxPPS API ver. 1 registered [ 0.832008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.835057] PTP clock support registered [ 0.838111] EDAC MC: Ver: 3.0.0 [ 0.841150] PCI: Using ACPI for IRQ routing [ 0.843656] NetLabel: Initializing [ 0.846015] NetLabel: domain hash size = 128 [ 0.847008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.848090] NetLabel: unlabeled traffic allowed by default [ 0.851124] vgaarb: loaded [ 0.853126] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.854010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.859091] clocksource: Switched to clocksource kvm-clock [ 0.962283] VFS: Disk quotas dquot_6.6.0 [ 0.963737] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.967211] *** VALIDATE ramfs *** [ 0.968703] *** VALIDATE hugetlbfs *** [ 0.970502] pnp: PnP ACPI init [ 0.972833] pnp: PnP ACPI: found 6 devices [ 0.989052] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.992537] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.995401] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.997898] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.000565] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.003546] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.006686] NET: Registered protocol family 2 [ 1.009253] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.013820] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.018040] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.023397] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.028284] TCP: Hash tables configured (established 65536 bind 65536) [ 1.032192] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.035883] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.039448] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.042861] NET: Registered protocol family 1 [ 1.045810] RPC: Registered named UNIX socket transport module. [ 1.047931] RPC: Registered udp transport module. [ 1.049665] RPC: Registered tcp transport module. [ 1.050714] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.052540] NET: Registered protocol family 44 [ 1.054044] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.056073] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.057751] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.060266] PCI: CLS 0 bytes, default 64 [ 1.062476] Unpacking initramfs... [ 2.453575] debug: unmapping init [mem 0xffff91c3fcc54000-0xffff91c3fffbffff] [ 2.459631] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.462400] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.466231] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229855db145, max_idle_ns: 440795257019 ns [ 2.955062] Initialise system trusted keyrings [ 2.957035] Key type blacklist registered [ 2.958885] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.967862] zbud: loaded [ 2.970808] *** VALIDATE nfs *** [ 2.972467] *** VALIDATE nfs4 *** [ 2.974117] pstore: using deflate compression [ 2.977350] Platform Keyring initialized [ 3.068870] NET: Registered protocol family 38 [ 3.070896] Key type asymmetric registered [ 3.072592] Asymmetric key parser 'x509' registered [ 3.074755] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.078110] io scheduler mq-deadline registered [ 3.079832] io scheduler kyber registered [ 3.081654] io scheduler bfq registered [ 3.083319] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.086483] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.090311] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.093679] ACPI: Power Button [PWRF] [ 3.099727] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.107308] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.118304] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.124366] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.140465] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.168396] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.196969] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.201824] Non-volatile memory driver v1.3 [ 3.203123] Linux agpgart interface v0.103 [ 3.233069] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.236424] vda: detected capacity change from 0 to 34836480 [ 3.252549] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.256219] vdb: detected capacity change from 0 to 1073741824 [ 3.270808] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.275374] vdc: detected capacity change from 0 to 2621440000 [ 3.289077] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.292686] vdd: detected capacity change from 0 to 2621440000 [ 3.307311] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.309820] vde: detected capacity change from 0 to 4294967296 [ 3.326486] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.329770] vdf: detected capacity change from 0 to 4294967296 [ 3.335511] libphy: Fixed MDIO Bus: probed [ 3.342318] usbcore: registered new interface driver usbserial_generic [ 3.345650] usbserial: USB Serial support registered for generic [ 3.348420] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.354969] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.357191] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.360677] mousedev: PS/2 mouse device common for all mice [ 3.363858] rtc_cmos 00:05: RTC can wake from S4 [ 3.366849] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.368452] rtc_cmos 00:05: registered as rtc0 [ 3.371460] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.372877] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.376260] intel_pstate: CPU model not supported [ 3.380709] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.383934] hid: raw HID events driver (C) Jiri Kosina [ 3.385779] usbcore: registered new interface driver usbhid [ 3.388348] usbhid: USB HID core driver [ 3.390195] drop_monitor: Initializing network drop monitor service [ 3.392880] Initializing XFRM netlink socket [ 3.395309] NET: Registered protocol family 10 [ 3.397970] Segment Routing with IPv6 [ 3.399593] NET: Registered protocol family 17 [ 3.402107] mpls_gso: MPLS GSO support [ 3.407912] RAS: Correctable Errors collector initialized. [ 3.410542] AVX version of gcm_enc/dec engaged. [ 3.412609] AES CTR mode by8 optimization enabled [ 3.492374] sched_clock: Marking stable (3492343242, 0)->(4400234560, -907891318) [ 3.495688] registered taskstats version 1 [ 3.497589] Loading compiled-in X.509 certificates [ 3.499938] zswap: loaded using pool lzo/zbud [ 3.522780] Key type big_key registered [ 3.535847] Key type encrypted registered [ 3.537561] ima: No TPM chip found, activating TPM-bypass! [ 3.539724] ima: Allocated hash algorithm: sha1 [ 3.541118] ima: No architecture policies found [ 3.542482] evm: Initialising EVM extended attributes: [ 3.543818] evm: security.selinux [ 3.544758] evm: security.ima [ 3.546013] evm: security.capability [ 3.546884] evm: HMAC attrs: 0x1 [ 3.549318] rtc_cmos 00:05: setting system clock to 2026-04-14 15:02:02 UTC (1776178922) [ 3.555743] debug: unmapping init [mem 0xffffffff8d603000-0xffffffff8d7fffff] [ 3.559120] debug: unmapping init [mem 0xffffffff8c382000-0xffffffff8c658fff] [ 3.568153] Write protecting the kernel read-only data: 28672k [ 3.571515] debug: unmapping init [mem 0xffffffff8aa03000-0xffffffff8abfffff] [ 3.574869] debug: unmapping init [mem 0xffffffff8b314000-0xffffffff8b3fffff] [ 3.605921] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.615359] systemd[1]: Detected virtualization kvm. [ 3.617850] systemd[1]: Detected architecture x86-64. [ 3.619842] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.651701] systemd[1]: No hostname configured. [ 3.653481] systemd[1]: Set hostname to . [ 3.656678] random: systemd: uninitialized urandom read (16 bytes read) [ 3.659369] systemd[1]: Initializing machine ID from random generator. [ 3.705166] random: ln: uninitialized urandom read (6 bytes read) [ 3.797260] random: systemd: uninitialized urandom read (16 bytes read) [ 3.800214] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.806694] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.811821] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.423240] device-mapper: uevent: version 1.0.3 [ 4.425682] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ 4.662447] random: fast init done [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.152173] virtio_net virtio0 ens2: renamed from eth0 [ 5.209944] scsi host0: ata_piix [ 5.224072] scsi host1: ata_piix [ 5.225790] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.227921] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.734782] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.054341] random: crng init done [ 10.055988] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.550528] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.712664] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.979390] SELinux: Disabled at runtime. [ 12.035241] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.042172] systemd[1]: Detected virtualization kvm. [ 12.043709] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.510500] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.514585] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.520211] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.523423] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.526320] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.534570] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.539621] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ 12.617643] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.975587] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.246060] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.259441] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.384859] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.393864] EDAC sbridge: Ver: 1.1.2 [ 14.999814] Key type dns_resolver registered [ 15.305316] NFS: Registering the id_resolver key type [ 15.308060] Key type id_resolver registered [ 15.310082] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg321-server login: [ 42.028373] libcfs: loading out-of-tree module taints kernel. [ 42.128207] alg: No test for adler32 (adler32-zlib) [ 42.894087] Key type ._llcrypt registered [ 42.899900] Key type .llcrypt registered [ 42.998211] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_hostid [ 67.907683] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 69.137653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.996045] Lustre: Lustre: Build Version: 2.15.8_13_g05ba8a9 [ 70.950045] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 70.955327] LNet: Accept secure, port 988 [ 72.775163] Key type lgssc registered [ 73.979823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.453830] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 90.406528] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.528382] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.622344] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 114.210962] hrtimer: interrupt took 8951317 ns [ 115.365563] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 127.351050] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 137.483412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 137.538878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.701955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 138.718305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 138.780513] Lustre: lustre-MDT0000: new disk, initializing [ 138.845718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.862122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 142.094545] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 153.699040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.774815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.843704] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 153.867628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 153.872360] Lustre: Skipped 1 previous similar message [ 153.937294] Lustre: lustre-MDT0001: new disk, initializing [ 154.006210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 154.044957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 154.057760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 158.011217] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 169.049870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 169.105830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 169.112272] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 169.258410] Lustre: lustre-OST0000: new disk, initializing [ 169.261753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 169.316441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.985790] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 176.165792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 176.173816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 183.580169] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 183.674182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.684709] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 183.802143] Lustre: lustre-OST0001: new disk, initializing [ 183.813564] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 183.876681] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.016359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 184.023577] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 187.881638] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 196.726837] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 207.793927] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 216.173721] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing check_logdir /tmp/testlogs/ [ 219.544351] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing yml_node [ 222.878369] Lustre: DEBUG MARKER: Client: 2.15.8.13 [ 225.005427] Lustre: DEBUG MARKER: MDS: 2.15.8.13 [ 227.655141] Lustre: DEBUG MARKER: OSS: 2.15.8.13 [ 229.141952] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Apr 14 11:05:47 EDT 2026 [ 235.749651] Lustre: DEBUG MARKER: excepting tests: 59 [ 241.883424] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing check_config_client /mnt/lustre [ 263.810977] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 267.574563] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 271.943482] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 276.231555] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 11:06:34 (1776179194) [ 283.229076] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 284.876414] Lustre: Failing over lustre-MDT0000 [ 285.047991] Lustre: server umount lustre-MDT0000 complete [ 285.098745] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 286.688684] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.691997] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 286.700105] Lustre: Skipped 3 previous similar messages [ 286.708512] LustreError: Skipped 2 previous similar messages [ 290.174566] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.203802] LustreError: Skipped 1 previous similar message [ 292.265960] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.280057] LustreError: Skipped 4 previous similar messages [ 293.855155] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179205/real 1776179205] req@00000000aa04fd75 x1862458659620672/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179212 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 293.895379] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.348462] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.360956] LustreError: Skipped 4 previous similar messages [ 307.840915] LDISKFS-fs (dm-0): recovery complete [ 307.847507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 308.605927] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 308.618755] LustreError: Skipped 15 previous similar messages [ 310.245829] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff1ef8c5 to 0x67748da0ff1eff78 [ 310.273215] Lustre: MGC192.168.203.121@tcp: Connection restored to (at 0@lo) [ 310.515993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 311.789271] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 314.889391] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 315.911614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 315.996581] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 324.608191] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 326.058903] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 333.516970] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 11:07:31 (1776179251) [ 335.184894] Lustre: Failing over lustre-OST0000 [ 335.226404] Lustre: server umount lustre-OST0000 complete [ 336.363226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.377595] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.395324] LustreError: Skipped 6 previous similar messages [ 354.339668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 354.588782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 356.574552] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 356.629450] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 356.635409] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.121@tcp (at 0@lo) [ 356.652979] Lustre: Skipped 3 previous similar messages [ 356.674038] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 359.366984] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 368.861833] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 370.648340] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 379.810075] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 11:08:17 (1776179297) [ 387.353753] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 388.948191] Lustre: Failing over lustre-MDT0000 [ 389.054705] Lustre: server umount lustre-MDT0000 complete [ 390.624510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 390.631814] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.635524] Lustre: Skipped 3 previous similar messages [ 390.669980] LustreError: Skipped 13 previous similar messages [ 397.791258] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179309/real 1776179309] req@000000001edbc726 x1862458659650496/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179316 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 397.833095] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 409.031886] LDISKFS-fs (dm-0): recovery complete [ 409.035498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.180372] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff1eff78 to 0x67748da0ff1f09ab [ 414.192446] Lustre: MGC192.168.203.121@tcp: Connection restored to 192.168.203.121@tcp (at 0@lo) [ 414.200347] Lustre: Skipped 1 previous similar message [ 414.389464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.097875] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 419.810610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 419.814093] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 419.948149] Lustre: lustre-MDT0000: Denying connection for new client 6993bf96-cedf-436d-aff9-db39f0d2d60c (at 192.168.203.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 425.346240] Lustre: lustre-MDT0000: Denying connection for new client 6993bf96-cedf-436d-aff9-db39f0d2d60c (at 192.168.203.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 430.462912] Lustre: lustre-MDT0000: Denying connection for new client 6993bf96-cedf-436d-aff9-db39f0d2d60c (at 192.168.203.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 435.595173] Lustre: lustre-MDT0000: Denying connection for new client 6993bf96-cedf-436d-aff9-db39f0d2d60c (at 192.168.203.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 440.723118] Lustre: lustre-MDT0000: Denying connection for new client 6993bf96-cedf-436d-aff9-db39f0d2d60c (at 192.168.203.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:44 [ 450.946520] Lustre: lustre-MDT0000: Denying connection for new client 6993bf96-cedf-436d-aff9-db39f0d2d60c (at 192.168.203.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:34 [ 450.965915] Lustre: Skipped 1 previous similar message [ 471.422527] Lustre: lustre-MDT0000: Denying connection for new client 6993bf96-cedf-436d-aff9-db39f0d2d60c (at 192.168.203.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:13 [ 471.438105] Lustre: Skipped 3 previous similar messages [ 485.000126] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 485.004631] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 485.013754] Lustre: lustre-MDT0000: Recovery over after 1:06, of 2 clients 1 recovered and 1 was evicted. [ 485.013873] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 485.025927] Lustre: Skipped 2 previous similar messages [ 485.040971] Lustre: lustre-OST0000: deleting orphan objects from 0x0:86 to 0x0:129 [ 493.358262] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 11:10:11 (1776179411) [ 498.638049] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 500.303295] Lustre: Failing over lustre-MDT0000 [ 500.474564] Lustre: server umount lustre-MDT0000 complete [ 500.703741] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 500.708538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.716870] Lustre: Skipped 1 previous similar message [ 500.724780] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.733938] LustreError: Skipped 24 previous similar messages [ 522.716665] LDISKFS-fs (dm-0): recovery complete [ 522.721981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.836163] LustreError: 11-0: MGC192.168.203.121@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 522.840358] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.866788] LustreError: 3358:0:(import.c:702:ptlrpc_connect_import_locked()) already connecting [ 522.866892] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff1f09ab to 0x67748da0ff1f0db0 [ 522.891156] Lustre: MGC192.168.203.121@tcp: Connection restored to (at 0@lo) [ 523.093543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 526.970360] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 528.354978] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 528.783032] Lustre: lustre-MDT0000: Denying connection for new client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 528.805475] Lustre: Skipped 2 previous similar messages [ 594.000187] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 594.010509] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 594.026932] Lustre: lustre-MDT0000: Recovery over after 1:06, of 2 clients 1 recovered and 1 was evicted. [ 594.027527] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 594.046683] Lustre: Skipped 3 previous similar messages [ 594.076079] Lustre: lustre-OST0000: deleting orphan objects from 0x0:86 to 0x0:161 [ 604.051092] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 11:12:01 (1776179521) [ 611.200981] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 613.443403] Lustre: Failing over lustre-MDT0000 [ 613.980257] Lustre: server umount lustre-MDT0000 complete [ 614.882568] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 614.897301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.927099] Lustre: Skipped 3 previous similar messages [ 622.561673] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179534/real 1776179534] req@00000000495023a8 x1862458659703616/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179541 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 622.613500] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 629.120320] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.141052] LustreError: Skipped 45 previous similar messages [ 637.058786] LDISKFS-fs (dm-0): recovery complete [ 637.063710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.968976] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff1f0db0 to 0x67748da0ff1f1145 [ 640.158430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 640.189075] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 640.385508] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 643.649765] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 645.664863] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 645.694058] Lustre: lustre-OST0000: deleting orphan objects from 0x0:86 to 0x0:193 [ 651.886915] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 653.531284] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 662.857097] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 11:13:00 (1776179580) [ 671.974992] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 674.194889] Lustre: Failing over lustre-MDT0000 [ 674.398793] Lustre: server umount lustre-MDT0000 complete [ 676.327698] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 676.330527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.371422] Lustre: Skipped 6 previous similar messages [ 684.001120] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179595/real 1776179595] req@0000000027a21496 x1862458659721408/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179602 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 684.043745] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 696.607287] LDISKFS-fs (dm-0): recovery complete [ 696.609810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.413060] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff1f1145 to 0x67748da0ff1f1582 [ 701.432408] Lustre: MGC192.168.203.121@tcp: Connection restored to (at 0@lo) [ 701.438453] Lustre: Skipped 5 previous similar messages [ 701.670332] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 703.429919] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 706.039483] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 707.141052] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 707.175287] Lustre: lustre-OST0001: deleting orphan objects from 0x0:99 to 0x0:129 [ 707.176242] Lustre: lustre-OST0000: deleting orphan objects from 0x0:86 to 0x0:225 [ 713.875755] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 715.237903] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 721.781548] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 11:14:00 (1776179640) [ 729.240551] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 731.523986] Lustre: Failing over lustre-MDT0000 [ 731.671767] Lustre: server umount lustre-MDT0000 complete [ 732.641673] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 732.648258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 732.676559] Lustre: Skipped 2 previous similar messages [ 739.807519] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179651/real 1776179651] req@00000000fd708768 x1862458659738048/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179658 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 739.841931] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 755.096387] LDISKFS-fs (dm-0): recovery complete [ 755.101671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 756.201107] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff1f1582 to 0x67748da0ff1f1a9f [ 756.514052] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 758.034697] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 761.329811] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 761.933021] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 761.958464] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 761.961165] Lustre: lustre-OST0001: deleting orphan objects from 0x0:99 to 0x0:161 [ 769.998866] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 771.312907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 777.439697] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 11:14:55 (1776179695) [ 783.787785] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 785.415934] Lustre: Failing over lustre-MDT0000 [ 785.558510] Lustre: server umount lustre-MDT0000 complete [ 787.426820] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 787.427837] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 787.445824] Lustre: Skipped 3 previous similar messages [ 794.591144] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179706/real 1776179706] req@0000000030ef889a x1862458659753728/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179713 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 794.614613] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 806.278145] LDISKFS-fs (dm-0): recovery complete [ 806.283330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.002113] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff1f1a9f to 0x67748da0ff1f1fc3 [ 812.267039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 812.276434] Lustre: Skipped 2 previous similar messages [ 812.333345] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 812.423578] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 816.357578] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 817.756325] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 817.812947] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:193 [ 817.815849] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:289 [ 824.031643] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 825.163539] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 831.514307] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 11:15:49 (1776179749) [ 837.628583] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 850.335289] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179762/real 1776179762] req@000000003c61efa4 x1862458659770560/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179769 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 850.365662] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 857.412560] LDISKFS-fs (dm-0): recovery complete [ 857.417095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.815239] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff1f1fc3 to 0x67748da0ff1f24fc [ 867.833633] Lustre: MGC192.168.203.121@tcp: Connection restored to (at 0@lo) [ 867.837630] Lustre: Skipped 14 previous similar messages [ 867.992907] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 871.476349] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 873.509704] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:225 [ 873.511210] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:321 [ 879.078517] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 880.582194] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 887.605529] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 11:16:45 (1776179805) [ 888.355894] Lustre: *** cfs_fail_loc=13b, val=315*** [ 888.358966] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 888.361792] LustreError: 6260:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000450a8d00 x1862458636570880/t38654705666(0) o35->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:33/0 lens 392/456 e 0 to 0 dl 1776179813 ref 1 fl Interpret:/0/0 rc 0/0 job:'openfile.0' [ 894.891555] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnecting [ 894.915165] Lustre: 6260:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000f4ddfd6e x1862458636570880/t38654705666(0) o35->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:39/0 lens 392/456 e 0 to 0 dl 1776179819 ref 1 fl Interpret:/2/0 rc 0/0 job:'openfile.0' [ 894.994432] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 896.479038] Lustre: Failing over lustre-MDT0000 [ 896.481632] Lustre: Skipped 1 previous similar message [ 896.614021] Lustre: server umount lustre-MDT0000 complete [ 896.616160] Lustre: Skipped 1 previous similar message [ 899.039630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.041582] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 899.041800] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 899.041805] LustreError: Skipped 170 previous similar messages [ 899.047339] Lustre: Skipped 7 previous similar messages [ 905.183113] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179817/real 1776179817] req@00000000f9b43def x1862458659787712/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179824 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 918.447501] LDISKFS-fs (dm-0): recovery complete [ 918.453727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.940829] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 923.520524] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 923.528793] Lustre: Skipped 1 previous similar message [ 927.355910] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 928.307035] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 928.324413] Lustre: Skipped 1 previous similar message [ 928.391661] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:353 [ 928.393462] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:257 [ 937.926605] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 939.784248] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 948.215176] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 11:17:46 (1776179866) [ 955.013352] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 958.948773] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 965.087174] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 965.101472] LustreError: Skipped 1 previous similar message [ 977.593767] LDISKFS-fs (dm-0): recovery complete [ 977.605447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.527132] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff1f29d3 to 0x67748da0ff1f2e87 [ 982.537103] Lustre: Skipped 1 previous similar message [ 982.854860] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 987.948301] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 988.340761] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:385 [ 988.343363] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:289 [ 1000.348888] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1002.583881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1011.011539] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 11:18:49 (1776179929) [ 1016.819838] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1017.705340] Lustre: *** cfs_fail_loc=114, val=0*** [ 1031.007140] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179942/real 1776179942] req@00000000f60c4935 x1862458659824512/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179949 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 1031.026219] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1040.134019] LDISKFS-fs (dm-0): recovery complete [ 1040.136536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.707449] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1051.301676] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1053.271050] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:417 [ 1053.274529] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:321 [ 1060.662574] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1062.321618] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1070.073881] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 11:19:48 (1776179988) [ 1076.276900] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1077.025140] Lustre: *** cfs_fail_loc=128, val=0*** [ 1079.383457] Lustre: Failing over lustre-MDT0000 [ 1079.387637] Lustre: Skipped 2 previous similar messages [ 1079.582498] Lustre: server umount lustre-MDT0000 complete [ 1079.590724] Lustre: Skipped 2 previous similar messages [ 1083.872677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.873496] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1083.887721] Lustre: Skipped 8 previous similar messages [ 1104.055249] LDISKFS-fs (dm-0): recovery complete [ 1104.058501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.147613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.154633] Lustre: Skipped 4 previous similar messages [ 1109.486343] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1109.502564] Lustre: Skipped 2 previous similar messages [ 1112.167280] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1113.663403] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1113.678337] Lustre: Skipped 2 previous similar messages [ 1113.756174] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:353 [ 1113.761305] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:449 [ 1122.737276] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1124.476194] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1132.438238] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 11:20:50 (1776180050) [ 1139.830734] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1151.455483] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1151.463214] LustreError: Skipped 2 previous similar messages [ 1161.820519] LDISKFS-fs (dm-0): recovery complete [ 1161.825193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.867959] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff1f3820 to 0x67748da0ff1f3e4e [ 1168.872523] Lustre: Skipped 2 previous similar messages [ 1168.876772] Lustre: MGC192.168.203.121@tcp: Connection restored to (at 0@lo) [ 1168.878858] Lustre: Skipped 24 previous similar messages [ 1169.074867] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1169.081117] Lustre: Skipped 1 previous similar message [ 1172.923529] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1174.771365] Lustre: lustre-OST0000: deleting orphan objects from 0x0:455 to 0x0:481 [ 1174.774331] Lustre: lustre-OST0001: deleting orphan objects from 0x0:359 to 0x0:385 [ 1181.105455] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1182.447412] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1189.965281] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 11:21:48 (1776180108) [ 1196.170512] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1207.775163] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180119/real 1776180119] req@000000000dc8f627 x1862458659877248/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180126 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 1207.805172] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1219.876117] LDISKFS-fs (dm-0): recovery complete [ 1219.882317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.685020] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1229.965763] Lustre: lustre-OST0000: deleting orphan objects from 0x0:487 to 0x0:513 [ 1229.971156] Lustre: lustre-OST0001: deleting orphan objects from 0x0:391 to 0x0:417 [ 1236.664131] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1237.864963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1245.450604] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 11:22:43 (1776180163) [ 1252.728398] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1282.484255] LDISKFS-fs (dm-0): recovery complete [ 1282.487738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.204162] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1292.832935] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:641 [ 1292.833032] Lustre: lustre-OST0001: deleting orphan objects from 0x0:528 to 0x0:545 [ 1298.679567] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1301.751966] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1328.312324] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 11:24:06 (1776180246) [ 1335.626969] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1337.394306] Lustre: Failing over lustre-MDT0000 [ 1337.396996] Lustre: Skipped 3 previous similar messages [ 1337.664781] Lustre: server umount lustre-MDT0000 complete [ 1337.667761] Lustre: Skipped 3 previous similar messages [ 1338.849505] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1340.895923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.909376] Lustre: Skipped 18 previous similar messages [ 1360.118459] LDISKFS-fs (dm-0): recovery complete [ 1360.124070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.786638] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1364.792744] Lustre: Skipped 2 previous similar messages [ 1369.096108] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1370.155544] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 1370.165765] Lustre: Skipped 3 previous similar messages [ 1370.219732] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:673 [ 1370.224115] Lustre: lustre-OST0001: deleting orphan objects from 0x0:528 to 0x0:577 [ 1378.294649] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1379.671892] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1389.509946] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 11:25:07 (1776180307) [ 1395.663974] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1412.582884] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.610189] LustreError: Skipped 310 previous similar messages [ 1420.735903] LDISKFS-fs (dm-0): recovery complete [ 1420.738957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.878439] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff1fd6cd to 0x67748da0ff1fdbf8 [ 1424.903689] Lustre: Skipped 3 previous similar messages [ 1425.280818] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1425.286601] Lustre: Skipped 4 previous similar messages [ 1430.358275] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1430.653635] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:705 [ 1430.656169] Lustre: lustre-OST0001: deleting orphan objects from 0x0:528 to 0x0:609 [ 1439.585332] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1441.097857] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1448.569802] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 11:26:06 (1776180366) [ 1454.934084] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1468.896807] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180380/real 1776180380] req@000000006af7f4cd x1862458659982272/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180387 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 1468.933025] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1468.940675] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1468.951396] LustreError: Skipped 4 previous similar messages [ 1478.435294] LDISKFS-fs (dm-0): recovery complete [ 1478.438580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.502804] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1491.044359] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:737 [ 1491.047928] Lustre: lustre-OST0001: deleting orphan objects from 0x0:528 to 0x0:641 [ 1499.072487] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1500.667997] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1508.555719] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 11:27:06 (1776180426) [ 1514.957802] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1537.362094] LDISKFS-fs (dm-0): recovery complete [ 1537.365676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.602480] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1551.017372] Lustre: lustre-OST0001: deleting orphan objects from 0x0:528 to 0x0:673 [ 1551.018965] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:769 [ 1558.869980] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1560.506944] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1567.782486] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 11:28:06 (1776180486) [ 1575.446934] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1601.791372] LDISKFS-fs (dm-0): recovery complete [ 1601.798867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1610.520448] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1611.322889] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:801 [ 1611.327988] Lustre: lustre-OST0001: deleting orphan objects from 0x0:528 to 0x0:705 [ 1620.655986] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1622.699801] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1631.135237] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 11:29:09 (1776180549) [ 1638.278653] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1641.952392] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1641.960255] LustreError: Skipped 2 previous similar messages [ 1661.863414] LDISKFS-fs (dm-0): recovery complete [ 1661.869298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.080708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.088749] Lustre: Skipped 8 previous similar messages [ 1662.128105] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1662.135301] Lustre: Skipped 4 previous similar messages [ 1666.971460] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1667.653412] Lustre: lustre-OST0001: deleting orphan objects from 0x0:528 to 0x0:737 [ 1667.657523] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:833 [ 1677.219753] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1678.845244] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1686.172629] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 11:30:04 (1776180604) [ 1693.445725] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1717.248995] LDISKFS-fs (dm-0): recovery complete [ 1717.252640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.350368] Lustre: MGC192.168.203.121@tcp: Connection restored to (at 0@lo) [ 1722.356987] Lustre: Skipped 44 previous similar messages [ 1726.504539] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1728.139272] Lustre: lustre-OST0000: deleting orphan objects from 0x0:835 to 0x0:865 [ 1728.144953] Lustre: lustre-OST0001: deleting orphan objects from 0x0:739 to 0x0:769 [ 1735.222925] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1736.807104] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1744.003493] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 11:31:02 (1776180662) [ 1750.000351] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1775.949502] LDISKFS-fs (dm-0): recovery complete [ 1775.955156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.977447] Lustre: lustre-OST0001: deleting orphan objects from 0x0:739 to 0x0:801 [ 1782.979416] Lustre: lustre-OST0000: deleting orphan objects from 0x0:835 to 0x0:897 [ 1783.820488] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1795.371433] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1797.280929] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1805.852497] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 11:32:03 (1776180723) [ 1812.416481] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1835.603639] LDISKFS-fs (dm-0): recovery complete [ 1835.607218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.497716] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1847.868134] Lustre: lustre-OST0001: deleting orphan objects from 0x0:803 to 0x0:833 [ 1847.868502] Lustre: lustre-OST0000: deleting orphan objects from 0x0:835 to 0x0:929 [ 1855.132683] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1856.760983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1864.326038] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 11:33:02 (1776180782) [ 1870.760449] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1872.569591] Lustre: Failing over lustre-MDT0000 [ 1872.573305] Lustre: Skipped 8 previous similar messages [ 1872.765562] Lustre: server umount lustre-MDT0000 complete [ 1872.772828] Lustre: Skipped 8 previous similar messages [ 1873.375861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.383017] Lustre: Skipped 32 previous similar messages [ 1896.848627] LDISKFS-fs (dm-0): recovery complete [ 1896.852400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.656615] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1903.669785] Lustre: Skipped 8 previous similar messages [ 1903.694266] Lustre: lustre-OST0000: deleting orphan objects from 0x0:835 to 0x0:961 [ 1903.697220] Lustre: lustre-OST0001: deleting orphan objects from 0x0:835 to 0x0:865 [ 1903.854401] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1913.828451] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1915.727923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1924.506754] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 11:34:02 (1776180842) [ 1931.658612] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1958.594167] LDISKFS-fs (dm-0): recovery complete [ 1958.598217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.889592] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff200733 to 0x67748da0ff200c8f [ 1958.904028] Lustre: Skipped 8 previous similar messages [ 1960.228360] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1960.240073] Lustre: Skipped 8 previous similar messages [ 1963.813616] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1964.682494] Lustre: lustre-OST0000: deleting orphan objects from 0x0:963 to 0x0:993 [ 1964.682732] Lustre: lustre-OST0001: deleting orphan objects from 0x0:867 to 0x0:897 [ 1972.354676] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1974.252476] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1981.907724] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 11:34:59 (1776180899) [ 1988.431125] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2002.399140] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180914/real 1776180914] req@00000000b6245455 x1862458660139264/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180921 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 2002.415203] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 2002.419698] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2002.425515] LustreError: Skipped 8 previous similar messages [ 2011.602159] LDISKFS-fs (dm-0): recovery complete [ 2011.606773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.054203] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.084808] LustreError: Skipped 363 previous similar messages [ 2024.236861] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2025.543149] Lustre: lustre-OST0001: deleting orphan objects from 0x0:899 to 0x0:929 [ 2025.543885] Lustre: lustre-OST0000: deleting orphan objects from 0x0:963 to 0x0:1025 [ 2033.557483] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2035.383815] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2044.886094] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 11:36:02 (1776180962) [ 2052.576664] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2076.432617] LDISKFS-fs (dm-0): recovery complete [ 2076.435736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.054837] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2086.431704] Lustre: lustre-OST0001: deleting orphan objects from 0x0:899 to 0x0:961 [ 2086.431819] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 2092.385804] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2093.544068] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2101.813071] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 11:36:59 (1776181019) [ 2109.930718] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2134.001597] LDISKFS-fs (dm-0): recovery complete [ 2134.006499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.988509] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2147.398474] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1059 to 0x0:1089 [ 2147.406062] Lustre: lustre-OST0001: deleting orphan objects from 0x0:963 to 0x0:993 [ 2151.469198] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2153.025072] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2160.808208] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 11:37:59 (1776181079) [ 2167.632711] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2191.469747] LDISKFS-fs (dm-0): recovery complete [ 2191.472647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.149907] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2198.155623] Lustre: Skipped 8 previous similar messages [ 2201.008397] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2203.711633] Lustre: lustre-OST0001: deleting orphan objects from 0x0:995 to 0x0:1025 [ 2203.712226] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1091 to 0x0:1121 [ 2207.870492] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2209.139825] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2215.916820] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:38:54 (1776181134) [ 2220.591973] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2224.097240] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2224.105920] LustreError: Skipped 6 previous similar messages [ 2248.427646] LDISKFS-fs (dm-0): recovery complete [ 2248.430702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.910151] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2254.450959] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1091 to 0x0:1153 [ 2254.458301] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 2262.785336] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2264.268959] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2273.080876] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 11:39:51 (1776181191) [ 2276.891466] Lustre: 77531:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 4209f5eb-68bc-4f03-bbd1-1242901d0815 at adminstrative request [ 2300.923711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.742461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.748532] Lustre: Skipped 10 previous similar messages [ 2312.044153] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2314.230889] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1089 [ 2314.231281] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 2318.050414] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2319.115688] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2324.063755] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2330.952244] Lustre: DEBUG MARKER: before 2844, after 2844 [ 2336.357616] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 11:40:54 (1776181254) [ 2337.721987] Lustre: 79562:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 4209f5eb-68bc-4f03-bbd1-1242901d0815 at adminstrative request [ 2349.291640] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:41:06 (1776181266) [ 2357.642939] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2379.028305] LDISKFS-fs (dm-0): recovery complete [ 2379.033179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.875972] Lustre: MGC192.168.203.121@tcp: Connection restored to (at 0@lo) [ 2384.886276] Lustre: Skipped 54 previous similar messages [ 2389.291982] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2390.646648] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1091 to 0x0:1121 [ 2390.647526] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1188 to 0x0:1217 [ 2396.791262] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2398.068809] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2404.652873] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:42:03 (1776181323) [ 2409.889669] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2431.937848] LDISKFS-fs (dm-0): recovery complete [ 2431.942777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.753278] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2445.859516] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2445.859769] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1123 to 0x0:1153 [ 2451.423935] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2452.611996] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2458.214924] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:42:56 (1776181376) [ 2464.258489] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2485.824811] LDISKFS-fs (dm-0): recovery complete [ 2485.827871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.947787] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2496.563419] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 2496.564835] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2501.553717] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2502.848983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2509.579443] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 11:43:47 (1776181427) [ 2515.162375] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2516.656774] Lustre: Failing over lustre-MDT0000 [ 2516.658449] Lustre: Skipped 10 previous similar messages [ 2516.770357] Lustre: server umount lustre-MDT0000 complete [ 2516.772521] Lustre: Skipped 10 previous similar messages [ 2516.959550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.970892] Lustre: Skipped 43 previous similar messages [ 2538.370596] LDISKFS-fs (dm-0): recovery complete [ 2538.373617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2547.684915] LustreError: 87048:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.121@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000ccc03abd) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2550.618533] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2553.325726] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2553.331283] Lustre: Skipped 10 previous similar messages [ 2553.360539] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 2553.365445] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1283 to 0x0:1313 [ 2557.539641] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2558.761886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2564.952657] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:44:43 (1776181483) [ 2570.088493] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2593.306844] LDISKFS-fs (dm-0): recovery complete [ 2593.312186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.349620] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff20433f to 0x67748da0ff20486a [ 2597.356064] Lustre: Skipped 10 previous similar messages [ 2598.794154] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2598.807776] Lustre: Skipped 10 previous similar messages [ 2600.720431] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2603.029338] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1283 to 0x0:1345 [ 2603.030048] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2608.586404] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2610.169617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2618.570503] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:45:36 (1776181536) [ 2625.095677] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2628.577327] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.598593] LustreError: Skipped 406 previous similar messages [ 2635.727568] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181547/real 1776181547] req@000000002353c117 x1862458660325824/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181554 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 2635.759033] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 2635.767065] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2635.775821] LustreError: Skipped 10 previous similar messages [ 2649.454767] LDISKFS-fs (dm-0): recovery complete [ 2649.458181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.215369] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2657.949215] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 2657.950872] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2664.518084] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2666.095112] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2674.149701] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:46:32 (1776181592) [ 2680.959137] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2705.130098] LDISKFS-fs (dm-0): recovery complete [ 2705.137068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.396681] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2713.744850] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1283 to 0x0:1313 [ 2713.748988] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 2719.949817] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2721.237611] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2728.876486] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:47:27 (1776181647) [ 2735.697366] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2760.356747] LDISKFS-fs (dm-0): recovery complete [ 2760.362178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.643667] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2769.527967] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2769.529522] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 2777.156466] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2778.735233] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2786.528726] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:48:24 (1776181704) [ 2793.890822] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2819.301742] LDISKFS-fs (dm-0): recovery complete [ 2819.305507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.890180] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2824.892784] Lustre: Skipped 10 previous similar messages [ 2828.621971] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2830.370318] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 2830.370318] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 2836.391901] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2837.769836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2844.587839] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:49:22 (1776181762) [ 2850.636294] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2873.079096] LDISKFS-fs (dm-0): recovery complete [ 2873.081662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2884.125272] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2886.166829] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 2886.166831] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1475 to 0x0:1505 [ 2891.371409] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2892.566179] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2899.349914] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 11:50:17 (1776181817) [ 2906.450912] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2911.718564] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2911.725024] LustreError: Skipped 7 previous similar messages [ 2929.094653] LDISKFS-fs (dm-0): recovery complete [ 2929.096587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2935.517684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.530848] Lustre: Skipped 10 previous similar messages [ 2939.269887] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2940.977815] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1507 to 0x0:1537 [ 2940.978066] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 2947.723926] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2949.272960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2956.616484] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 11:51:14 (1776181874) [ 2957.538926] Lustre: 102244:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 4209f5eb-68bc-4f03-bbd1-1242901d0815 at adminstrative request [ 2966.758387] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 11:51:24 (1776181884) [ 2972.178774] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2983.952609] LDISKFS-fs (dm-0): recovery complete [ 2983.956975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.055791] Lustre: MGC192.168.203.121@tcp: Connection restored to (at 0@lo) [ 2990.065981] Lustre: Skipped 54 previous similar messages [ 2990.269392] LustreError: 103525:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 2990.276158] LustreError: 103525:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2990.285561] Lustre: 103559:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2990.292727] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2990.299190] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2990.316645] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2990.358672] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1539 to 0x0:1569 [ 2990.359240] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 2993.038328] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2995.684317] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3011.845064] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 11:52:10 (1776181930) [ 3018.405951] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3030.235642] LDISKFS-fs (dm-0): recovery complete [ 3030.245145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.599948] LustreError: 105454:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.121@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000393f1c1e) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3034.848453] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3034.868926] LustreError: 105444:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3034.878697] LustreError: 105444:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3034.891881] Lustre: 105478:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3034.898207] Lustre: 105478:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3034.905172] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3034.913046] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3034.937671] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3034.999840] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1601 [ 3035.000790] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1479 to 0x0:1505 [ 3038.412375] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3040.238277] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3051.190706] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3051.196200] Lustre: Skipped 1 previous similar message [ 3056.697893] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 11:52:54 (1776181974) [ 3063.605116] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3074.442814] LDISKFS-fs (dm-0): recovery complete [ 3074.448824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.740068] LustreError: 107363:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3074.747561] LustreError: 107363:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3074.754756] Lustre: 107396:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3074.760265] Lustre: 107396:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3074.765818] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3074.775548] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3074.790534] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3074.831074] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1512 to 0x0:1537 [ 3074.831510] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1607 to 0x0:1633 [ 3078.180734] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3080.174356] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3098.764784] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 11:53:36 (1776182016) [ 3100.098621] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3100.101051] LustreError: 6258:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000069fd4b7c x1862458637101888/t201863462916(0) o36->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:735/0 lens 504/456 e 0 to 0 dl 1776182025 ref 1 fl Interpret:/0/0 rc 0/0 job:'rm.0' [ 3113.873190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.458496] LustreError: 108739:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3119.467573] LustreError: 108739:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3119.478102] Lustre: 108773:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3119.490114] Lustre: 108773:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3119.496896] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3119.532109] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3119.564180] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3119.620838] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1512 to 0x0:1569 [ 3119.623150] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1635 to 0x0:1665 [ 3124.076793] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3124.705712] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3144.148908] Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ============ 11:54:22 (1776182062) [ 3150.396296] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3152.103894] Lustre: Failing over lustre-MDT0000 [ 3152.108949] Lustre: Skipped 11 previous similar messages [ 3152.259357] Lustre: server umount lustre-MDT0000 complete [ 3152.261733] Lustre: Skipped 11 previous similar messages [ 3155.424439] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.446957] Lustre: Skipped 48 previous similar messages [ 3172.281963] LDISKFS-fs (dm-0): recovery complete [ 3172.285382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.414153] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3185.702229] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3185.709921] Lustre: Skipped 7 previous similar messages [ 3185.743142] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1571 to 0x0:1601 [ 3185.744612] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1635 to 0x0:1697 [ 3194.529325] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 11:55:12 (1776182112) [ 3202.330603] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3216.832795] LDISKFS-fs (dm-0): recovery complete [ 3216.836163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.425193] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff208b99 to 0x67748da0ff2090ee [ 3219.437433] Lustre: Skipped 11 previous similar messages [ 3219.612974] LustreError: 112503:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3219.618410] LustreError: 112503:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3219.623716] Lustre: 112536:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3219.628139] Lustre: 112536:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3219.632641] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3219.638116] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3219.651483] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3219.689905] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1635 to 0x0:1729 [ 3219.690026] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1571 to 0x0:1633 [ 3223.202701] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3225.063886] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3241.797130] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:56:00 (1776182160) [ 3269.028959] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3275.653365] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.671983] LustreError: Skipped 376 previous similar messages [ 3283.936137] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182195/real 1776182195] req@000000008976db3b x1862458660553152/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182202 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 3283.966113] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 3283.987187] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3284.003290] LustreError: Skipped 11 previous similar messages [ 3293.569681] LDISKFS-fs (dm-0): recovery complete [ 3293.575195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.590338] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3301.604032] Lustre: Skipped 7 previous similar messages [ 3305.112834] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3306.178853] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2130 to 0x0:2145 [ 3306.179338] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2034 to 0x0:2049 [ 3313.863817] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3315.111273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3333.657953] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:57:31 (1776182251) [ 3359.350416] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3393.633927] LDISKFS-fs (dm-0): recovery complete [ 3393.639202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3400.501163] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3406.113873] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2546 to 0x0:2561 [ 3406.114261] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2465 [ 3411.616724] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3413.400031] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3432.222731] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 11:59:10 (1776182350) [ 3433.544755] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 3435.084330] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 11:59:13 (1776182353) [ 3437.117402] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3438.015750] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3438.024964] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3438.055043] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2497 [ 3443.479259] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 11:59:21 (1776182361) [ 3463.247348] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3494.247798] LDISKFS-fs (dm-2): recovery complete [ 3494.253058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3494.447452] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3494.452361] Lustre: Skipped 20 previous similar messages [ 3496.318263] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2963 to 0x0:3009 [ 3496.331134] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3500.147179] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3556.307636] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 12:01:14 (1776182474) [ 3563.434360] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3566.477429] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.21@tcp (stopping) [ 3570.143796] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3570.147557] LustreError: Skipped 8 previous similar messages [ 3590.810523] LDISKFS-fs (dm-0): recovery complete [ 3590.817562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3595.752985] Lustre: MGC192.168.203.121@tcp: Connection restored to 192.168.203.121@tcp (at 0@lo) [ 3595.768520] Lustre: Skipped 42 previous similar messages [ 3596.074563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3596.088040] Lustre: Skipped 9 previous similar messages [ 3601.513136] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3601.514344] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2913 [ 3601.805876] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3611.294809] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3613.123350] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3632.315058] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 12:02:30 (1776182550) [ 3636.758935] LustreError: 6257:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3641.823310] LustreError: 6257:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3641.830980] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnecting [ 3641.929759] LustreError: 10024:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3643.918912] LustreError: 6258:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3646.432028] LustreError: 122362:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3646.435991] LustreError: 8229:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3646.473972] LustreError: 8229:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 3646.485108] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3646.485427] LustreError: 6258:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=2450 [ 3647.523970] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2963 to 0x0:3041 [ 3648.315634] LustreError: 6258:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3653.599109] LustreError: 6258:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3653.606379] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnecting [ 3653.612361] Lustre: Skipped 1 previous similar message [ 3655.334718] LustreError: 6258:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3660.768419] LustreError: 6258:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3660.788198] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnecting [ 3662.554521] LustreError: 27225:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3667.935326] LustreError: 27225:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3674.594585] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnecting [ 3674.605383] Lustre: Skipped 1 previous similar message [ 3676.329548] LustreError: 27225:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3676.349991] LustreError: 27225:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message [ 3681.759108] LustreError: 27225:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3681.762981] LustreError: 27225:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message [ 3696.095153] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnecting [ 3696.103472] Lustre: Skipped 2 previous similar messages [ 3697.933972] LustreError: 13422:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3697.945926] LustreError: 13422:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages [ 3703.263086] LustreError: 13422:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3703.274687] LustreError: 13422:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages [ 3713.528720] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 12:03:51 (1776182631) [ 3715.354879] LustreError: 6257:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3720.583388] Lustre: lustre-MDT0000: Export 000000001a72dd2c already connecting from 192.168.203.21@tcp [ 3722.108138] Lustre: lustre-MDT0000: Export 000000001a72dd2c already connecting from 192.168.203.21@tcp [ 3723.847348] Lustre: lustre-MDT0000: Export 000000001a72dd2c already connecting from 192.168.203.21@tcp [ 3727.386626] Lustre: lustre-MDT0000: Export 000000001a72dd2c already connecting from 192.168.203.21@tcp [ 3727.397415] Lustre: Skipped 1 previous similar message [ 3732.969793] Lustre: lustre-MDT0000: Export 000000001a72dd2c already connecting from 192.168.203.21@tcp [ 3732.974312] Lustre: Skipped 3 previous similar messages [ 3738.447844] LustreError: 6257:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 3738.457035] Lustre: 6257:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/3s); client may timeout req@00000000f665fa44 x1862458638133568/t0(0) o38->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:0/0 lens 520/416 e 0 to 0 dl 1776182654 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0' [ 3740.030037] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnecting [ 3740.038200] Lustre: Skipped 2 previous similar messages [ 3743.136980] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 12:04:21 (1776182661) [ 3750.146500] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3753.762711] Lustre: Failing over lustre-MDT0000 [ 3753.765764] Lustre: Skipped 5 previous similar messages [ 3753.918439] Lustre: server umount lustre-MDT0000 complete [ 3753.921912] Lustre: Skipped 5 previous similar messages [ 3765.765474] LDISKFS-fs (dm-0): recovery complete [ 3765.768724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3766.039883] Lustre: *** cfs_fail_loc=712, val=0*** [ 3766.059745] LustreError: 8227:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery req@00000000a9366f17 x1862458660868288/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' [ 3766.100243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3766.174457] LustreError: 127039:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3766.182982] LustreError: 127039:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3766.189339] Lustre: 127073:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3766.195859] Lustre: 127073:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3766.201723] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3766.210617] Lustre: lustre-MDT0000-osd: cancel update llog [0x200018aa0:0x1:0x0] [ 3766.225918] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3766.271385] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2945 [ 3766.303200] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2963 to 0x0:3073 [ 3770.825286] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3771.362091] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3791.840339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.861472] Lustre: Skipped 26 previous similar messages [ 3805.286284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3811.049619] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3811.325677] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3811.338430] Lustre: Skipped 4 previous similar messages [ 3811.448989] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2963 to 0x0:3105 [ 3811.465873] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2977 [ 3827.574438] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3830.484424] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3843.109359] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 12:06:00 (1776182760) [ 3843.375683] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnecting [ 3851.498398] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:06:09 (1776182769) [ 3852.526442] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3852.530314] LustreError: 6265:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000004d24c700 x1862458638160192/t0(0) o700->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:732/0 lens 264/248 e 0 to 0 dl 1776182777 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0' [ 3877.856051] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3877.870593] LustreError: Skipped 191 previous similar messages [ 3881.199905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.282244] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff23bc2a to 0x67748da0ff23c0c9 [ 3881.287869] Lustre: Skipped 5 previous similar messages [ 3885.324922] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3886.657276] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3107 to 0x0:3137 [ 3886.657299] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2979 to 0x0:3009 [ 3894.908631] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3896.527184] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3907.332955] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:07:04 (1776182824) [ 3928.239828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3929.638445] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3929.652829] Lustre: Skipped 5 previous similar messages [ 3929.745595] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97 [ 3929.754643] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169 [ 3933.383270] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3943.428934] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3945.333407] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4016.890559] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:08:55 (1776182935) [ 4024.082130] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4037.087185] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182949/real 1776182949] req@00000000ab32ca75 x1862458660944256/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182956 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 4037.108562] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4037.119549] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4037.131688] LustreError: Skipped 5 previous similar messages [ 4047.219428] LDISKFS-fs (dm-0): recovery complete [ 4047.221507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.114319] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4060.637446] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3030 to 0x0:3073 [ 4060.643112] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3201 [ 4130.413282] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:10:48 (1776183048) [ 4131.803124] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4131.806970] Lustre: Skipped 2 previous similar messages [ 4131.813586] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3212 to 0x0:3233 [ 4132.611315] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3212 to 0x0:3265 [ 4142.762503] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:11:01 (1776183061) [ 4147.680303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4150.241081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4150.247940] Lustre: Skipped 4 previous similar messages [ 4168.970708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4180.254239] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4180.259621] Lustre: Skipped 8 previous similar messages [ 4184.490863] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4185.589794] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4185.594698] LustreError: 135187:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000f9c2f5c4 x1862458638215872/t0(0) o101->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:342/0 lens 328/344 e 0 to 0 dl 1776183142 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0' [ 4224.900290] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnected, waiting for 2 clients in recovery for 0:57 [ 4224.969737] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 4224.979543] Lustre: Skipped 34 previous similar messages [ 4225.001351] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3297 [ 4225.001795] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3084 to 0x0:3105 [ 4230.668105] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4232.125958] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4239.277018] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:12:37 (1776183157) [ 4241.233766] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4248.087300] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4250.592098] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4250.600256] LustreError: Skipped 6 previous similar messages [ 4250.607568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4272.623383] LDISKFS-fs (dm-0): recovery complete [ 4272.627962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4276.974132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4276.991939] Lustre: Skipped 6 previous similar messages [ 4281.702587] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4282.425943] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3329 [ 4282.426359] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3137 [ 4290.768926] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4292.589506] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4299.971209] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:13:38 (1776183218) [ 4300.877208] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4307.958440] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4329.775907] LDISKFS-fs (dm-0): recovery complete [ 4329.780454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4342.467449] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4343.316326] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3169 [ 4343.321169] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3361 [ 4351.803619] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4353.485537] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4361.459261] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:14:39 (1776183279) [ 4362.617152] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4372.617271] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4375.194437] Lustre: Failing over lustre-MDT0000 [ 4375.199523] Lustre: Skipped 7 previous similar messages [ 4375.497644] Lustre: server umount lustre-MDT0000 complete [ 4375.502147] Lustre: Skipped 7 previous similar messages [ 4398.893578] LDISKFS-fs (dm-0): recovery complete [ 4398.895870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.628438] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4409.404597] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3201 [ 4409.405098] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3393 [ 4420.182661] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:15:38 (1776183338) [ 4422.392440] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4422.395872] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4422.401721] LustreError: 6260:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000c9606eab x1862458638246400/t261993005072(0) o35->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:547/0 lens 392/456 e 0 to 0 dl 1776183347 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4429.792951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4429.817141] Lustre: Skipped 31 previous similar messages [ 4444.677931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4457.616538] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4459.007648] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4459.016799] Lustre: Skipped 7 previous similar messages [ 4459.038373] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3425 [ 4459.038734] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3233 [ 4459.084205] Lustre: 6260:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000c19bf936 x1862458638246400/t261993005072(0) o35->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:583/0 lens 392/456 e 0 to 0 dl 1776183383 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4466.101674] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4467.353768] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4474.639613] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:16:32 (1776183392) [ 4475.748653] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4475.755212] LustreError: 6257:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000006e75f5f5 x1862458638254784/t266287972368(0) o36->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:635/0 lens 504/448 e 0 to 0 dl 1776183435 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4483.717964] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4485.546247] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.560252] LustreError: Skipped 256 previous similar messages [ 4505.613898] LDISKFS-fs (dm-0): recovery complete [ 4505.619215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4513.258496] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff23fcc0 to 0x67748da0ff240350 [ 4513.266542] Lustre: Skipped 6 previous similar messages [ 4516.994754] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4518.941646] Lustre: 27225:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000076a59d2a x1862458638254784/t266287972368(0) o36->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:678/0 lens 504/448 e 0 to 0 dl 1776183478 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4518.942080] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3457 [ 4518.942888] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3265 [ 4524.598792] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4525.791957] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4532.929822] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:17:31 (1776183451) [ 4534.063904] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4534.072059] LustreError: 13422:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000005337a026 x1862458638263424/t270582939664(0) o36->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:694/0 lens 504/448 e 0 to 0 dl 1776183494 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4535.832547] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4541.971882] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4566.046542] LDISKFS-fs (dm-0): recovery complete [ 4566.052580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.894557] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4569.905350] Lustre: Skipped 7 previous similar messages [ 4573.238074] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4574.736828] Lustre: 43045:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000002b38549c x1862458638263808/t270582939665(0) o35->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:699/0 lens 392/456 e 0 to 0 dl 1776183499 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4574.736926] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3489 [ 4574.739456] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3267 to 0x0:3297 [ 4574.771625] Lustre: 43045:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4584.128365] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:18:22 (1776183502) [ 4585.098555] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4585.101792] Lustre: Skipped 1 previous similar message [ 4585.106101] LustreError: 13422:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000075ec0277 x1862458638271168/t274877906960(0) o36->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:745/0 lens 504/448 e 0 to 0 dl 1776183545 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4585.128069] LustreError: 13422:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 4586.857255] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4586.867199] Lustre: Skipped 1 previous similar message [ 4593.269491] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4594.046879] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnecting [ 4594.062561] Lustre: Skipped 3 previous similar messages [ 4594.076363] Lustre: 6257:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000008b0d4ca3 x1862458638271168/t274877906960(0) o36->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:754/0 lens 504/448 e 0 to 0 dl 1776183554 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4616.094497] LDISKFS-fs (dm-0): recovery complete [ 4616.098727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4623.678726] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4625.445375] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3521 [ 4625.449671] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3329 [ 4633.638662] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:19:11 (1776183551) [ 4635.088028] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4637.058683] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4637.060722] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4637.064627] LustreError: 6260:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000a7bd9e8d x1862458638278912/t279172874256(0) o35->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:7/0 lens 392/456 e 0 to 0 dl 1776183562 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4644.303853] Lustre: 6260:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000a2675ab3 x1862458638278912/t279172874256(0) o35->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:14/0 lens 392/456 e 0 to 0 dl 1776183569 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4644.701141] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4657.119337] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183569/real 1776183569] req@000000002b9de9e5 x1862458661121344/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183576 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 4657.130068] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 4657.133548] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4657.145973] LustreError: Skipped 8 previous similar messages [ 4669.445771] LDISKFS-fs (dm-0): recovery complete [ 4669.451473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.965870] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4680.235182] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3331 to 0x0:3361 [ 4680.251641] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3553 [ 4687.771974] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:20:06 (1776183606) [ 4688.409430] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4688.414152] LustreError: 6258:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000b6dbea18 x1862458638285568/t283467841549(0) o101->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:93/0 lens 664/600 e 0 to 0 dl 1776183648 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0' [ 4732.821087] Lustre: 6257:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000010bb4c41 x1862458638285568/t283467841549(0) o101->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:137/0 lens 664/3424 e 0 to 0 dl 1776183692 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0' [ 4738.727797] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:20:56 (1776183656) [ 4744.804667] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4766.408736] LDISKFS-fs (dm-0): recovery complete [ 4766.413806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.803974] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4781.112657] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3393 [ 4781.118021] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3585 [ 4786.894224] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4788.294334] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4805.523764] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:22:03 (1776183723) [ 4812.260275] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4834.855125] LDISKFS-fs (dm-0): recovery complete [ 4834.860720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.419296] Lustre: MGC192.168.203.121@tcp: Connection restored to (at 0@lo) [ 4840.428313] Lustre: Skipped 45 previous similar messages [ 4840.603722] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4840.607145] Lustre: Skipped 9 previous similar messages [ 4843.606489] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4846.128956] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3425 [ 4846.129709] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3587 to 0x0:3617 [ 4852.004088] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4853.523486] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4859.107553] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4871.219322] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:23:09 (1776183789) [ 4916.053914] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4922.848106] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4922.859385] LustreError: Skipped 6 previous similar messages [ 4938.641532] LDISKFS-fs (dm-0): recovery complete [ 4938.647539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4946.557484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4946.564840] Lustre: Skipped 9 previous similar messages [ 4950.856910] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4952.424465] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4676 to 0x0:4705 [ 4952.425092] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4868 to 0x0:4897 [ 4959.237474] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4960.521710] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5023.954282] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:25:42 (1776183942) [ 5030.712503] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5032.190381] Lustre: Failing over lustre-MDT0000 [ 5032.195140] Lustre: Skipped 8 previous similar messages [ 5032.371121] Lustre: server umount lustre-MDT0000 complete [ 5032.374923] Lustre: Skipped 8 previous similar messages [ 5033.953061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5033.964315] Lustre: Skipped 33 previous similar messages [ 5052.744357] LDISKFS-fs (dm-0): recovery complete [ 5052.747887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.607934] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5063.262383] Lustre: lustre-MDT0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 5063.272617] Lustre: Skipped 7 previous similar messages [ 5063.309447] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4676 to 0x0:4737 [ 5063.314976] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4899 to 0x0:4929 [ 5070.516382] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5071.951358] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5082.445758] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5083.941332] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5089.895058] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:26:47 (1776184007) [ 5097.349826] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5138.336080] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnecting [ 5138.349573] Lustre: Skipped 2 previous similar messages [ 5140.620211] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5140.622785] Lustre: Skipped 1 previous similar message [ 5140.630376] LustreError: 6258:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000d76acf27 x1862458639773312/t300647710728(0) o36->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:544/0 lens 66040/440 e 0 to 0 dl 1776184099 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0' [ 5183.366943] Lustre: 27225:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000daab2a5a x1862458639773312/t300647710728(0) o36->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:587/0 lens 66040/440 e 0 to 0 dl 1776184142 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0' [ 5191.543843] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5192.805944] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:28:31 (1776184111) [ 5203.873294] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5206.499323] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5206.509105] LustreError: Skipped 316 previous similar messages [ 5228.644919] LDISKFS-fs (dm-0): recovery complete [ 5228.649878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5235.170429] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff282e67 to 0x67748da0ff285ce3 [ 5235.180664] Lustre: Skipped 7 previous similar messages [ 5235.896380] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5235.903451] Lustre: Skipped 6 previous similar messages [ 5238.274734] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5241.492695] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4838 to 0x0:4865 [ 5241.492783] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5031 to 0x0:5057 [ 5245.867391] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5247.447673] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5256.322611] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:29:34 (1776184174) [ 5277.382278] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5311.516156] LDISKFS-fs (dm-2): recovery complete [ 5311.523608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5314.809307] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5328.181837] LustreError: 166322:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 5328.190734] Lustre: 165804:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5328.204250] Lustre: 165804:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 5328.210138] Lustre: 165804:0:(ldlm_lib.c:1804:abort_req_replay_queue()) @@@ aborted: req@000000005a2bcd3b x1862458661639808/t0(17179870574) o6->lustre-MDT0000-mdtlov_UUID@0@lo:720/0 lens 544/0 e 0 to 0 dl 1776184275 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'osp-syn-0-0.0' [ 5328.229489] LustreError: 165804:0:(ofd_obd.c:1309:ofd_iocontrol()) lustre-OST0000: not supported cmd = 0xc00866c1 [ 5328.232074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5328.239959] Lustre: 165804:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 5347.275164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5352.293552] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5357.536417] Lustre: 3354:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184232/real 1776184232] req@0000000011a6910d x1862458661689664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776184276 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 5357.554395] Lustre: 3354:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 5357.585428] LustreError: 3354:0:(client.c:3161:ptlrpc_replay_interpret()) @@@ status 0, old was -19 req@00000000e90e8ca1 x1862458661639808/t17179870574(17179870574) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 1776184312 ref 2 fl Interpret:RQU/4/0 rc 0/0 job:'osp-syn-0-0.0' [ 5358.306819] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5458 to 0x0:5473 [ 5358.335888] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129 [ 5363.920563] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5365.257153] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5403.930209] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:32:02 (1776184322) [ 5406.188949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5418.975121] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5418.982506] LustreError: Skipped 5 previous similar messages [ 5423.728323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5427.498221] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5429.799207] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5266 to 0x0:5281 [ 5429.807312] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5458 to 0x0:5505 [ 5441.415161] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.21@tcp (stopping) [ 5441.422821] Lustre: Skipped 3 previous similar messages [ 5458.053862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5458.406485] Lustre: MGC192.168.203.121@tcp: Connection restored to (at 0@lo) [ 5458.415071] Lustre: Skipped 26 previous similar messages [ 5458.594638] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5458.607827] Lustre: Skipped 6 previous similar messages [ 5462.068551] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5464.080594] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5266 to 0x0:5313 [ 5464.081047] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5458 to 0x0:5537 [ 5469.493321] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5470.768478] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5477.604657] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:33:15 (1776184395) [ 5508.834871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5510.231248] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161 [ 5510.232146] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5569 [ 5512.521638] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5520.826901] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5522.412412] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5530.374253] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:34:08 (1776184448) [ 5539.831319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5539.912305] Lustre: *** cfs_fail_loc=605, val=0*** [ 5539.915115] LustreError: 172388:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=0000000062c28237 failed: rc = -95 [ 5539.920273] LustreError: 172388:0:(obd_config.c:774:class_setup()) setup MGS failed (-95) [ 5539.925044] LustreError: 172388:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95 [ 5539.930236] LustreError: 172388:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered [ 5539.934804] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5539.941121] LustreError: 172388:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 5539.949974] LustreError: 172388:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5546.455542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5548.172907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5548.178950] Lustre: Skipped 7 previous similar messages [ 5551.808583] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5553.701914] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5315 to 0x0:5345 [ 5553.710132] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5601 [ 5557.556988] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 12:34:36 (1776184476) [ 5563.486727] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5568.999663] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5569.008697] LustreError: Skipped 12 previous similar messages [ 5589.003168] LDISKFS-fs (dm-0): recovery complete [ 5589.006473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.967059] Lustre: *** cfs_fail_loc=707, val=0*** [ 5596.771815] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5633.962339] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 5634.572977] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5358 to 0x0:5377 [ 5634.573812] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5615 to 0x0:5633 [ 5640.561211] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5642.206545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5651.037898] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 12:36:09 (1776184569) [ 5679.429258] LustreError: 27225:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms [ 5685.471865] LustreError: 27225:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5702.081342] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 12:37:00 (1776184620) [ 5732.636424] LustreError: 8233:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms [ 5738.687330] LustreError: 8233:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake [ 5746.483908] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 12:37:44 (1776184664) [ 5774.238654] LustreError: 6257:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms [ 5779.239115] LustreError: 6257:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5781.338890] LustreError: 6256:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms [ 5791.439291] LustreError: 6256:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5809.417754] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 12:38:47 (1776184727) [ 5867.251419] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 12:39:45 (1776184785) [ 5896.472341] LustreError: 6256:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 5896.903186] LustreError: 6256:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5904.729551] LustreError: 6260:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 5904.745734] LustreError: 6260:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 5905.183217] LustreError: 6260:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5905.199573] LustreError: 6260:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 5920.735996] LustreError: 8231:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 5920.741775] LustreError: 8231:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 63 previous similar messages [ 5921.600310] LustreError: 6256:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5921.614128] LustreError: 6256:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 65 previous similar messages [ 5930.305072] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 12:40:48 (1776184848) [ 5966.366645] Lustre: DEBUG MARKER: phase 2 [ 5976.517805] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 12:41:34 (1776184894) [ 6058.744246] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 12:42:57 (1776184977) [ 6060.271800] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6062.307497] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 12:43:00 (1776184980) [ 6066.949270] Lustre: DEBUG MARKER: Started rundbench load pid=138642 ... [ 6075.409780] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6078.346799] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6080.175975] Lustre: Failing over lustre-MDT0000 [ 6080.178807] Lustre: Skipped 8 previous similar messages [ 6080.314266] Lustre: server umount lustre-MDT0000 complete [ 6080.319414] Lustre: Skipped 9 previous similar messages [ 6080.606592] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6080.620171] LustreError: Skipped 163 previous similar messages [ 6082.530385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.543272] Lustre: Skipped 25 previous similar messages [ 6093.090934] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185003/real 1776185003] req@00000000a7882305 x1862458661935168/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776185010 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 6093.117360] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 6093.134139] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6093.148421] LustreError: Skipped 3 previous similar messages [ 6103.314744] LDISKFS-fs (dm-0): recovery complete [ 6103.329137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6110.692421] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff297ca7 to 0x67748da0ff2a7e5e [ 6110.711040] Lustre: Skipped 4 previous similar messages [ 6110.722324] Lustre: MGC192.168.203.121@tcp: Connection restored to (at 0@lo) [ 6110.730521] Lustre: Skipped 16 previous similar messages [ 6111.010563] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6111.019122] Lustre: Skipped 3 previous similar messages [ 6112.637502] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6112.647905] Lustre: Skipped 7 previous similar messages [ 6115.947398] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6116.418051] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6116.429284] Lustre: Skipped 8 previous similar messages [ 6116.468296] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5426 to 0x0:5441 [ 6116.468750] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5702 to 0x0:5729 [ 6127.950191] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6130.444234] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6141.845897] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6144.421185] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6146.918810] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.21@tcp (stopping) [ 6169.965938] LDISKFS-fs (dm-1): recovery complete [ 6169.969634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6170.228458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6170.233617] Lustre: Skipped 2 previous similar messages [ 6174.834614] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6176.962632] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:319 to 0x280000400:353 [ 6176.963500] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:157 to 0x2c0000400:193 [ 6184.075480] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6185.598791] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6217.645275] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 12:45:35 (1776185135) [ 6345.811442] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6357.977694] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 6359.550318] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.21@tcp (stopping) [ 6359.560699] Lustre: Skipped 3 previous similar messages [ 6361.571944] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6361.577588] LustreError: Skipped 2 previous similar messages [ 6383.321767] LDISKFS-fs (dm-1): recovery complete [ 6383.325138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6387.749495] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6391.534895] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:537 to 0x2c0000400:577 [ 6391.536997] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:696 to 0x280000400:737 [ 6397.196474] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6398.924136] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6459.253437] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 12:49:37 (1776185377) [ 6587.837267] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6600.012471] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 6601.701978] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.21@tcp (stopping) [ 6624.409198] LDISKFS-fs (dm-0): recovery complete [ 6624.422392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6632.954412] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6635.686301] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6142 to 0x0:6177 [ 6635.688577] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5855 to 0x0:5889 [ 6643.341150] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6645.000942] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6652.507968] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 12:52:50 (1776185570) [ 6781.166373] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6793.293670] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 6795.163390] Lustre: Failing over lustre-MDT0001 [ 6795.165216] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6795.174711] Lustre: Skipped 3 previous similar messages [ 6795.190487] Lustre: Skipped 13 previous similar messages [ 6795.198547] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6795.333385] Lustre: server umount lustre-MDT0001 complete [ 6795.345934] Lustre: Skipped 3 previous similar messages [ 6795.746336] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6795.756710] LustreError: Skipped 113 previous similar messages [ 6819.536771] LDISKFS-fs (dm-1): recovery complete [ 6819.540792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6819.846879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6819.850863] Lustre: Skipped 2 previous similar messages [ 6819.881954] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6819.886939] Lustre: Skipped 3 previous similar messages [ 6821.629085] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6821.638256] Lustre: Skipped 3 previous similar messages [ 6823.645595] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6824.936380] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 6824.941085] Lustre: Skipped 15 previous similar messages [ 6824.966085] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6824.970586] Lustre: Skipped 3 previous similar messages [ 6825.023191] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:707 to 0x2c0000400:737 [ 6825.024433] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:868 to 0x280000400:897 [ 6834.189719] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6835.968899] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6844.709702] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 12:56:02 (1776185762) [ 6855.390533] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6857.807175] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 6858.185128] Lustre: lustre-OST0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnecting [ 6858.202976] Lustre: Skipped 1 previous similar message [ 6885.182954] LDISKFS-fs (dm-2): recovery complete [ 6885.192873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6887.607912] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6689 to 0x0:6721 [ 6887.616286] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:868 to 0x280000400:929 [ 6890.729919] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6902.200342] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6904.745514] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6914.872520] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 12:57:13 (1776185833) [ 7045.927670] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7053.318206] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7066.400748] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 7068.729569] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.21@tcp (stopping) [ 7074.117586] LustreError: 44521:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776185993 with bad export cookie 7454739005820902614 [ 7074.118748] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7074.144712] LustreError: 44521:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7074.154535] LustreError: Skipped 1 previous similar message [ 7081.951199] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185993/real 1776185993] req@00000000468adca2 x1862458664832640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776186000 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 7082.003855] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7096.732967] LDISKFS-fs (dm-0): recovery complete [ 7096.739690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7107.625044] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff341cd6 to 0x67748da0ff3b74fe [ 7107.644566] Lustre: Skipped 1 previous similar message [ 7112.534871] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7137.551140] LDISKFS-fs (dm-1): recovery complete [ 7137.555883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7138.194868] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 7138.207529] LustreError: Skipped 2 previous similar messages [ 7143.706742] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:868 to 0x280000400:961 [ 7143.709384] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:707 to 0x2c0000400:769 [ 7143.873788] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7152.770188] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6401 to 0x0:6433 [ 7152.770188] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6689 to 0x0:6753 [ 7158.577045] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7160.558078] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7162.252118] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7170.450677] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 13:01:28 (1776186088) [ 7177.439410] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7200.015265] LDISKFS-fs (dm-0): recovery complete [ 7200.019463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7209.741152] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7214.209892] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7217.066313] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnected, waiting for 2 clients in recovery for 0:57 [ 7217.123170] Lustre: 195572:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000ba74747e x1862458646150144/t330712487276(330712487276) o101->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:322/0 lens 648/3424 e 0 to 0 dl 1776186142 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7217.306048] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6401 to 0x0:6465 [ 7217.306542] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6755 to 0x0:6785 [ 7227.121986] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7229.941739] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7238.968921] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 13:02:36 (1776186156) [ 7246.065435] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7269.382552] LDISKFS-fs (dm-0): recovery complete [ 7269.385795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7273.896937] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7273.906087] LustreError: 195572:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000d6dcc4ab x1862458646150144/t330712487276(330712487276) o101->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:378/0 lens 648/600 e 0 to 0 dl 1776186198 ref 1 fl Interpret:/4/0 rc 301/0 job:'lfs.0' [ 7279.190023] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7281.045641] Lustre: lustre-MDT0000: Client 4209f5eb-68bc-4f03-bbd1-1242901d0815 (at 192.168.203.21@tcp) reconnected, waiting for 2 clients in recovery for 0:57 [ 7281.068464] Lustre: 195571:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000006db6b510 x1862458646150144/t330712487276(330712487276) o101->4209f5eb-68bc-4f03-bbd1-1242901d0815@192.168.203.21@tcp:386/0 lens 648/3424 e 0 to 0 dl 1776186206 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7281.229736] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:6817 [ 7281.231684] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6401 to 0x0:6497 [ 7291.453995] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7292.988739] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7301.424924] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 13:03:39 (1776186219) [ 7333.543724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7343.699343] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7345.171833] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6401 to 0x0:6529 [ 7351.164701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7352.567616] Lustre: lustre-OST0000: Denying connection for new client e999958f-c5e2-4f3a-a5f4-294786d35f8a (at 192.168.203.21@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7352.582313] Lustre: Skipped 12 previous similar messages [ 7353.171557] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:6849 [ 7353.183711] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:868 to 0x280000400:993 [ 7355.433900] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7365.185219] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 13:04:43 (1776186283) [ 7366.282084] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7366.285219] LustreError: 185203:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000072a4a571 x1862458665071168/t347892350983(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:471/0 lens 1056/4320 e 0 to 0 dl 1776186291 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7372.594511] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7373.803193] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7397.352237] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7397.367422] LustreError: Skipped 209 previous similar messages [ 7398.929344] LDISKFS-fs (dm-0): recovery complete [ 7398.936385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7408.239608] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7409.231995] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:6881 [ 7409.232112] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6561 [ 7416.525992] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7418.045708] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7427.126237] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 13:05:44 (1776186344) [ 7428.080961] LustreError: 185203:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000001d1d7350 x1862458665092800/t352187318282(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:533/0 lens 2200/4320 e 0 to 0 dl 1776186353 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7434.998934] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7435.232211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7435.251452] Lustre: Skipped 28 previous similar messages [ 7435.261108] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7435.278993] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 7435.281658] Lustre: Skipped 33 previous similar messages [ 7441.695676] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7443.335793] Lustre: Failing over lustre-MDT0001 [ 7443.339623] Lustre: Skipped 8 previous similar messages [ 7444.969935] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7449.167955] Lustre: server umount lustre-MDT0001 complete [ 7449.171194] Lustre: Skipped 8 previous similar messages [ 7470.327936] LDISKFS-fs (dm-1): recovery complete [ 7470.330409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.553022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7470.557559] Lustre: Skipped 8 previous similar messages [ 7470.584550] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7470.589805] Lustre: Skipped 8 previous similar messages [ 7471.998669] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7472.006733] Lustre: Skipped 8 previous similar messages [ 7474.224260] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7475.699857] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7475.705419] Lustre: Skipped 8 previous similar messages [ 7475.728387] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:801 [ 7475.732327] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1025 [ 7482.201747] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7483.790486] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7491.374533] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 13:06:49 (1776186409) [ 7492.243186] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7492.247256] Lustre: Skipped 1 previous similar message [ 7492.250282] LustreError: 185208:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000d9a0ef1a x1862458665112128/t352187318301(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:597/0 lens 2200/4320 e 0 to 0 dl 1776186417 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7498.663188] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7499.744874] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7505.515991] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7528.846408] LDISKFS-fs (dm-0): recovery complete [ 7528.850350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7537.447312] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7539.859186] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6593 [ 7539.859612] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:6913 [ 7546.528944] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7548.379766] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7573.116676] LDISKFS-fs (dm-1): recovery complete [ 7573.128123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7578.108028] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7578.666358] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:812 to 0x2c0000400:833 [ 7578.683183] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1057 [ 7588.727692] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7590.653471] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7598.944948] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:08:37 (1776186517) [ 7606.241305] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7609.188614] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7615.898128] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7620.339029] LustreError: 6243:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186539 with bad export cookie 7454739005821421986 [ 7620.345668] LustreError: 6243:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7621.510984] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.21@tcp (stopping) [ 7621.516982] Lustre: Skipped 4 previous similar messages [ 7650.112575] LDISKFS-fs (dm-0): recovery complete [ 7650.120113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7658.735295] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7684.703333] LDISKFS-fs (dm-1): recovery complete [ 7684.715948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7689.704473] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7690.410403] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:844 to 0x2c0000400:865 [ 7690.415793] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1068 to 0x280000400:1089 [ 7690.500803] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6625 [ 7690.502575] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:6945 [ 7699.158117] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7700.783462] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7702.538546] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7712.399985] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:10:30 (1776186630) [ 7713.531722] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7713.537980] Lustre: Skipped 1 previous similar message [ 7713.543165] LustreError: 214089:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000008f08c47 x1862458647032128/t34359738438(0) o36->e999958f-c5e2-4f3a-a5f4-294786d35f8a@192.168.203.21@tcp:63/0 lens 560/448 e 0 to 0 dl 1776186638 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 7713.561684] LustreError: 214089:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 7719.818882] Lustre: lustre-MDT0001: Client e999958f-c5e2-4f3a-a5f4-294786d35f8a (at 192.168.203.21@tcp) reconnecting [ 7719.842133] Lustre: 214643:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000ca40f125 x1862458647032128/t34359738438(0) o36->e999958f-c5e2-4f3a-a5f4-294786d35f8a@192.168.203.21@tcp:69/0 lens 560/448 e 0 to 0 dl 1776186644 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 7725.214441] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7738.850210] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186650/real 1776186650] req@00000000c8748ac6 x1862458665183424/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776186657 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 7738.883177] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [ 7738.891854] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7738.905329] LustreError: Skipped 6 previous similar messages [ 7751.192784] LDISKFS-fs (dm-0): recovery complete [ 7751.205537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.234552] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff3c190e to 0x67748da0ff3c274d [ 7755.241049] Lustre: Skipped 6 previous similar messages [ 7759.491760] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7760.957168] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6657 [ 7760.960580] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:6977 [ 7768.494451] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7770.209971] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7778.912363] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:11:36 (1776186696) [ 7787.327429] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7787.406095] Lustre: lustre-MDT0001: Client e999958f-c5e2-4f3a-a5f4-294786d35f8a (at 192.168.203.21@tcp) reconnecting [ 7787.454599] Lustre: 214089:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000073884a8f x1862458647049664/t34359738508(0) o36->e999958f-c5e2-4f3a-a5f4-294786d35f8a@192.168.203.21@tcp:137/0 lens 560/448 e 0 to 0 dl 1776186712 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 7810.805200] LDISKFS-fs (dm-1): recovery complete [ 7810.807607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7814.975870] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7816.210686] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:886 to 0x2c0000400:929 [ 7816.211532] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1110 to 0x280000400:1153 [ 7822.875632] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7824.573753] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7832.297916] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:12:30 (1776186750) [ 7840.130863] Lustre: lustre-MDT0001: Client e999958f-c5e2-4f3a-a5f4-294786d35f8a (at 192.168.203.21@tcp) reconnecting [ 7840.149708] Lustre: 214643:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000643a5d98 x1862458647065664/t38654705734(0) o36->e999958f-c5e2-4f3a-a5f4-294786d35f8a@192.168.203.21@tcp:190/0 lens 560/448 e 0 to 0 dl 1776186765 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 7841.806666] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7847.274347] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7870.615583] LDISKFS-fs (dm-0): recovery complete [ 7870.632607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7880.460821] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7881.288129] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6689 [ 7881.290449] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:7009 [ 7892.497932] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7894.476855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7922.434650] LDISKFS-fs (dm-1): recovery complete [ 7922.438914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7927.967787] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7928.335637] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1164 to 0x280000400:1185 [ 7928.336479] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:940 to 0x2c0000400:961 [ 7937.941927] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7939.522470] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7949.753111] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:14:27 (1776186867) [ 7958.407470] Lustre: lustre-MDT0001: Client e999958f-c5e2-4f3a-a5f4-294786d35f8a (at 192.168.203.21@tcp) reconnecting [ 7958.437681] Lustre: 214090:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000005295c4d x1862458647087616/t42949673030(0) o36->e999958f-c5e2-4f3a-a5f4-294786d35f8a@192.168.203.21@tcp:308/0 lens 560/448 e 0 to 0 dl 1776186883 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 7962.765561] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7971.818935] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7973.774423] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.21@tcp (stopping) [ 7973.782056] Lustre: Skipped 4 previous similar messages [ 7974.374446] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7974.380812] LustreError: Skipped 10 previous similar messages [ 7978.001495] LustreError: 8246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186896 with bad export cookie 7454739005821435636 [ 7978.026658] LustreError: 8246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8002.005886] LDISKFS-fs (dm-0): recovery complete [ 8002.009757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.406889] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8010.425242] LustreError: Skipped 228 previous similar messages [ 8014.385070] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8042.611743] LDISKFS-fs (dm-1): recovery complete [ 8042.614770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8048.107511] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 8048.125428] Lustre: Skipped 39 previous similar messages [ 8048.696591] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:7041 [ 8048.697767] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6721 [ 8048.733462] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1196 to 0x280000400:1217 [ 8048.748490] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:972 to 0x2c0000400:993 [ 8049.574612] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8061.559053] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8064.802796] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8066.597445] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8076.734739] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:16:34 (1776186994) [ 8078.009778] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8078.021549] Lustre: Skipped 3 previous similar messages [ 8078.033247] LustreError: 185203:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000036b47744 x1862458665287360/t373662154767(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:427/0 lens 640/4320 e 0 to 0 dl 1776187002 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 8078.050371] LustreError: 185203:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 3 previous similar messages [ 8084.447335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8084.454882] Lustre: Skipped 36 previous similar messages [ 8084.472916] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8084.835232] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8086.656888] Lustre: Failing over lustre-MDT0001 [ 8086.660595] Lustre: Skipped 10 previous similar messages [ 8086.792665] Lustre: server umount lustre-MDT0001 complete [ 8086.798218] Lustre: Skipped 10 previous similar messages [ 8109.942810] LDISKFS-fs (dm-1): recovery complete [ 8109.950546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8110.241965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8110.246122] Lustre: Skipped 10 previous similar messages [ 8110.260298] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8110.264448] Lustre: Skipped 10 previous similar messages [ 8110.974982] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8110.994583] Lustre: Skipped 10 previous similar messages [ 8115.703249] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8115.711067] Lustre: Skipped 10 previous similar messages [ 8115.728172] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1249 [ 8115.728173] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1004 to 0x2c0000400:1025 [ 8116.101410] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8126.531675] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8128.204620] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8137.992320] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:17:35 (1776187055) [ 8146.412318] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8146.939557] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8174.099152] LDISKFS-fs (dm-0): recovery complete [ 8174.101622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.870901] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:7073 [ 8180.873991] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6753 [ 8180.926562] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8191.050459] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8192.769864] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8202.343596] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:18:39 (1776187119) [ 8210.403918] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8210.718276] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8217.681594] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8243.440038] LDISKFS-fs (dm-0): recovery complete [ 8243.446655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8250.037903] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6785 [ 8250.052398] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:7105 [ 8250.815632] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8261.400332] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8262.952275] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8291.667883] LDISKFS-fs (dm-1): recovery complete [ 8291.684507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8297.130292] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8297.500852] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1281 [ 8297.500903] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1004 to 0x2c0000400:1057 [ 8307.155319] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8309.458290] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8320.576377] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:20:38 (1776187238) [ 8329.700184] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8331.686709] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8340.634992] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8346.843173] LustreError: 6241:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187265 with bad export cookie 7454739005821445961 [ 8346.857697] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8346.885805] LustreError: Skipped 4 previous similar messages [ 8357.279142] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187269/real 1776187269] req@0000000095e1f709 x1862458665369088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776187276 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 8357.294675] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 8370.286523] LDISKFS-fs (dm-0): recovery complete [ 8370.289718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8379.744676] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff3c6749 to 0x67748da0ff3c70c6 [ 8379.753275] Lustre: Skipped 4 previous similar messages [ 8385.549273] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8410.783902] LDISKFS-fs (dm-1): recovery complete [ 8410.785870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8415.475190] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8416.391211] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1313 [ 8416.391212] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1004 to 0x2c0000400:1089 [ 8416.512165] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6817 [ 8416.513640] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:7137 [ 8427.996906] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8430.137724] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8433.093963] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8443.669691] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:22:41 (1776187361) [ 8452.483345] Lustre: lustre-MDT0001: Client e999958f-c5e2-4f3a-a5f4-294786d35f8a (at 192.168.203.21@tcp) reconnecting [ 8452.494477] Lustre: 239122:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000bfb4d6a8 x1862458647147136/t60129542146(0) o36->e999958f-c5e2-4f3a-a5f4-294786d35f8a@192.168.203.21@tcp:47/0 lens 496/2888 e 0 to 0 dl 1776187377 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0' [ 8453.418082] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8479.391697] LDISKFS-fs (dm-0): recovery complete [ 8479.394820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8486.017273] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8487.013558] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6849 [ 8487.016896] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:7169 [ 8495.346941] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8497.344885] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8505.495481] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 13:23:43 (1776187423) [ 8514.089938] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8540.811744] LDISKFS-fs (dm-1): recovery complete [ 8540.818802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8546.370826] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1345 [ 8546.371619] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1004 to 0x2c0000400:1121 [ 8546.688304] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8559.712668] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8562.239346] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8571.225020] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 13:24:49 (1776187489) [ 8580.538543] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8588.515789] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8592.351748] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8592.360802] LustreError: Skipped 7 previous similar messages [ 8610.787888] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8610.811836] LustreError: Skipped 244 previous similar messages [ 8613.753088] LDISKFS-fs (dm-0): recovery complete [ 8613.756205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8621.564260] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8622.678979] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:7201 [ 8622.679193] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6881 [ 8631.068628] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8632.892648] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8659.487070] LDISKFS-fs (dm-1): recovery complete [ 8659.490850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8665.025775] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8665.064458] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 8665.071472] Lustre: Skipped 43 previous similar messages [ 8665.117436] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1004 to 0x2c0000400:1153 [ 8665.119708] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1377 [ 8674.534341] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8676.606048] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8685.458611] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 13:26:43 (1776187603) [ 8686.732382] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8686.741365] Lustre: Skipped 6 previous similar messages [ 8686.745678] LustreError: 237723:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000c47c7ec6 x1862458647175744/t68719476738(0) o36->e999958f-c5e2-4f3a-a5f4-294786d35f8a@192.168.203.21@tcp:281/0 lens 496/456 e 0 to 0 dl 1776187611 ref 1 fl Interpret:/0/0 rc 0/0 job:'rmdir.0' [ 8686.770756] LustreError: 237723:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 6 previous similar messages [ 8692.874345] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8694.185962] Lustre: lustre-MDT0001: Client e999958f-c5e2-4f3a-a5f4-294786d35f8a (at 192.168.203.21@tcp) reconnecting [ 8694.196222] Lustre: Skipped 2 previous similar messages [ 8699.751843] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8701.819905] Lustre: Failing over lustre-MDT0000 [ 8701.824714] Lustre: Skipped 9 previous similar messages [ 8703.978250] Lustre: server umount lustre-MDT0000 complete [ 8703.983092] Lustre: Skipped 9 previous similar messages [ 8706.018201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8706.033721] Lustre: Skipped 40 previous similar messages [ 8707.534065] LustreError: 44518:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187626 with bad export cookie 7454739005821453437 [ 8707.543955] LustreError: 44518:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8732.126606] LDISKFS-fs (dm-0): recovery complete [ 8732.130947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8741.247676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8741.250161] Lustre: Skipped 9 previous similar messages [ 8741.299275] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8741.309075] Lustre: Skipped 9 previous similar messages [ 8746.004520] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8756.618326] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8756.625313] Lustre: Skipped 9 previous similar messages [ 8769.586788] LDISKFS-fs (dm-1): recovery complete [ 8769.592711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8774.851574] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8776.231241] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8776.242495] Lustre: Skipped 9 previous similar messages [ 8776.287856] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1004 to 0x2c0000400:1185 [ 8776.293823] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1409 [ 8776.331233] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6913 [ 8776.332548] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6787 to 0x0:7233 [ 8785.875548] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8787.652992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8789.344932] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8797.016585] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 13:28:35 (1776187715) [ 8798.584528] Lustre: 251600:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting e999958f-c5e2-4f3a-a5f4-294786d35f8a at adminstrative request [ 8808.549066] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 13:28:46 (1776187726) [ 8834.772948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8839.788783] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8841.034324] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6964 to 0x0:7009 [ 8841.039584] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7285 to 0x0:7329 [ 8849.064330] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8850.533963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8858.888183] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 13:29:36 (1776187776) [ 8894.832453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8896.673739] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7430 to 0x0:7457 [ 8896.679938] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1441 [ 8899.632576] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8909.662434] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8911.458931] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8919.212391] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 13:30:37 (1776187837) [ 8923.106041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8929.075508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8932.145951] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8933.658205] Lustre: lustre-MDT0000: Denying connection for new client 9bbdf327-61de-4fd0-8bd7-2f7398db05e1 (at 192.168.203.21@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 8934.931633] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6964 to 0x0:7041 [ 8934.932039] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7430 to 0x0:7489 [ 8945.527050] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 13:31:03 (1776187863) [ 8953.318728] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8978.569182] LDISKFS-fs (dm-2): recovery complete [ 8978.575724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8980.720292] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7491 to 0x0:7521 [ 8980.728099] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1473 [ 8982.892869] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8991.262958] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8992.896773] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9002.973072] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 13:32:00 (1776187920) [ 9012.124886] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9037.038418] LDISKFS-fs (dm-2): recovery complete [ 9037.043387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9039.295104] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.21@tcp inode [0x20002ea31:0x5:0x0] object 0x0:7522 extent [0-4194303]: client csum 2ac5c5f3, server csum 4b208041 [ 9039.465259] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7523 to 0x0:7553 [ 9039.473935] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1505 [ 9040.920121] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9049.655081] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9051.203875] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9058.572439] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 13:32:56 (1776187976) [ 9065.042883] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9070.870374] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9079.595148] LustreError: 6241:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187998 with bad export cookie 7454739005821480149 [ 9079.595470] LustreError: 166-1: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9079.608807] LustreError: 6241:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9079.623754] LustreError: Skipped 5 previous similar messages [ 9080.801458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9102.425274] LDISKFS-fs (dm-0): recovery complete [ 9102.430144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9113.064266] Lustre: Evicted from MGS (at 192.168.203.121@tcp) after server handle changed from 0x67748da0ff3cecd5 to 0x67748da0ff3cfec5 [ 9113.072198] Lustre: Skipped 5 previous similar messages [ 9117.479582] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9119.586848] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6964 to 0x0:7073 [ 9141.934517] LDISKFS-fs (dm-2): recovery complete [ 9141.938926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9143.824418] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1537 [ 9146.957478] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9164.183159] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 13:34:42 (1776188082) [ 9185.247612] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776188097/real 1776188097] req@000000001b2b8998 x1862458665609920/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776188104 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 9185.274151] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 19 previous similar messages [ 9199.359847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9207.861499] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9208.343601] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6964 to 0x0:7105 [ 9213.427886] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9213.449336] LustreError: Skipped 261 previous similar messages [ 9217.158712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9222.948950] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9226.409752] Lustre: lustre-OST0000: Denying connection for new client f17105e6-9f54-4955-9859-b86098aa114f (at 192.168.203.21@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 9231.747069] Lustre: lustre-OST0000: Denying connection for new client f17105e6-9f54-4955-9859-b86098aa114f (at 192.168.203.21@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:57 [ 9236.863556] Lustre: lustre-OST0000: Denying connection for new client f17105e6-9f54-4955-9859-b86098aa114f (at 192.168.203.21@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 9247.107273] Lustre: lustre-OST0000: Denying connection for new client f17105e6-9f54-4955-9859-b86098aa114f (at 192.168.203.21@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:41 [ 9247.129445] Lustre: Skipped 1 previous similar message [ 9267.586625] Lustre: lustre-OST0000: Denying connection for new client f17105e6-9f54-4955-9859-b86098aa114f (at 192.168.203.21@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:21 [ 9267.605911] Lustre: Skipped 3 previous similar messages [ 9289.000156] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 9289.004263] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 9289.029272] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.121@tcp (at 0@lo) [ 9289.044520] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1569 [ 9289.046155] Lustre: Skipped 35 previous similar messages [ 9289.069508] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7604 to 0x0:7625 [ 9295.698923] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 61 sec [ 9303.795903] Lustre: DEBUG MARKER: free_before: 7646580 free_after: 7646580 [ 9309.582258] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 13:37:07 (1776188227) [ 9313.975996] Lustre: Failing over lustre-OST0001 [ 9313.978538] Lustre: Skipped 10 previous similar messages [ 9314.034459] Lustre: server umount lustre-OST0001 complete [ 9314.036626] Lustre: Skipped 10 previous similar messages [ 9314.274160] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9314.285710] LustreError: Skipped 13 previous similar messages [ 9314.291587] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9314.309197] Lustre: Skipped 26 previous similar messages [ 9332.239382] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9333.981413] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1004 to 0x2c0000400:1217 [ 9333.989812] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7108 to 0x0:7137 [ 9337.435200] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9347.764118] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 13:37:45 (1776188265) [ 9369.621273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9369.887541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9369.895567] Lustre: Skipped 11 previous similar messages [ 9369.904106] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 9369.910645] Lustre: Skipped 11 previous similar messages [ 9369.987184] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9369.989473] Lustre: Skipped 11 previous similar messages [ 9371.039336] Lustre: *** cfs_fail_loc=715, val=0*** [ 9371.935742] LustreError: 268259:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 40000ms [ 9371.943192] LustreError: 268259:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 9 previous similar messages [ 9372.960249] Lustre: *** cfs_fail_loc=715, val=0*** [ 9373.673618] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9378.272190] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:56 [ 9378.280176] Lustre: Skipped 1 previous similar message [ 9379.201897] Lustre: lustre-OST0000: Client f17105e6-9f54-4955-9859-b86098aa114f (at 192.168.203.21@tcp) reconnected, waiting for 3 clients in recovery for 0:55 [ 9379.295340] Lustre: *** cfs_fail_loc=715, val=0*** [ 9379.297532] Lustre: Skipped 2 previous similar messages [ 9385.440743] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:49 [ 9385.448804] Lustre: Skipped 1 previous similar message [ 9386.463297] Lustre: *** cfs_fail_loc=715, val=0*** [ 9386.467216] Lustre: Skipped 2 previous similar messages [ 9392.607901] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:42 [ 9392.620587] Lustre: Skipped 2 previous similar messages [ 9393.635205] Lustre: *** cfs_fail_loc=715, val=0*** [ 9393.640727] Lustre: Skipped 2 previous similar messages [ 9399.685326] Lustre: lustre-OST0000: Client f17105e6-9f54-4955-9859-b86098aa114f (at 192.168.203.21@tcp) reconnected, waiting for 3 clients in recovery for 0:35 [ 9399.700793] Lustre: Skipped 1 previous similar message [ 9407.903229] Lustre: *** cfs_fail_loc=715, val=0*** [ 9407.906126] Lustre: Skipped 5 previous similar messages [ 9411.943116] LustreError: 268259:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 9411.950864] LustreError: 268259:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 7 previous similar messages [ 9411.980661] Lustre: lustre-OST0000: Recovery over after 0:42, of 3 clients 3 recovered and 0 were evicted. [ 9411.990800] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7629 to 0x0:7657 [ 9412.001358] Lustre: Skipped 11 previous similar messages [ 9412.004287] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1601 [ 9418.467077] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9419.904939] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9427.891218] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 13:39:06 (1776188346) [ 9450.587385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9463.199927] Lustre: *** cfs_fail_loc=715, val=0*** [ 9463.202063] Lustre: Skipped 2 previous similar messages [ 9465.866346] LustreError: 269825:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 80000ms [ 9468.429328] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9469.316082] Lustre: lustre-MDT0000: Client f17105e6-9f54-4955-9859-b86098aa114f (at 192.168.203.21@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 9469.338891] Lustre: Skipped 5 previous similar messages [ 9472.991737] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9486.312469] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9486.329173] Lustre: Skipped 1 previous similar message [ 9489.793179] Lustre: lustre-MDT0000: Client f17105e6-9f54-4955-9859-b86098aa114f (at 192.168.203.21@tcp) reconnected, waiting for 2 clients in recovery for 0:38 [ 9489.804269] Lustre: Skipped 2 previous similar messages [ 9498.020249] Lustre: *** cfs_fail_loc=715, val=0*** [ 9498.023520] Lustre: Skipped 9 previous similar messages [ 9507.808622] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9507.815211] Lustre: Skipped 2 previous similar messages [ 9524.611366] Lustre: lustre-MDT0000: Client f17105e6-9f54-4955-9859-b86098aa114f (at 192.168.203.21@tcp) reconnected, waiting for 2 clients in recovery for 0:03 [ 9524.620400] Lustre: Skipped 4 previous similar messages [ 9531.774273] Lustre: lustre-MDT0000: Recovery already passed deadline 0:03. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 9538.942112] Lustre: lustre-MDT0000: Recovery already passed deadline 0:10. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 9542.625717] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9542.645039] Lustre: Skipped 4 previous similar messages [ 9545.887153] LustreError: 269825:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 9545.908471] Lustre: 269825:0:(ldlm_lib.c:2849:target_recovery_thread()) too long recovery - read logs [ 9545.915813] LustreError: dumping log to /tmp/lustre-log.1776188464.269825 [ 9546.085214] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7148 to 0x0:7169 [ 9546.087588] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7668 to 0x0:7689 [ 9553.350775] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9555.015972] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9564.587571] Lustre: DEBUG MARKER: == replay-single test complete, duration 9334 sec ======== 13:41:22 (1776188482) [ 9599.355835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.840419] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9608.727585] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7668 to 0x0:7721 [ 9608.728153] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7148 to 0x0:7201 [ 9617.495752] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9618.930413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9624.034587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9624.041552] Lustre: Skipped 3 previous similar messages [ 9638.619539] LustreError: 6243:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188557 with bad export cookie 7454739005821513049 [ 9638.635479] LustreError: 6243:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9675.097505] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 9677.451522] Key type lgssc unregistered [ 9677.790593] LNet: 274669:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9677.808984] LNet: Removed LNI 192.168.203.121@tcp [ 9678.609321] Key type .llcrypt unregistered [ 9678.612355] Key type ._llcrypt unregistered