[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 588910766 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002338] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007412] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010164] LSM: Security Framework initializing [ 0.012060] Yama: becoming mindful. [ 0.013047] SELinux: Initializing. [ 0.014078] *** VALIDATE selinux *** [ 0.022222] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027069] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028204] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031068] *** VALIDATE tmpfs *** [ 0.032511] *** VALIDATE proc *** [ 0.034010] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036302] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037217] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039039] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043787] debug: unmapping init [mem 0xffffffffbc859000-0xffffffffbc860fff] [ 0.047358] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048916] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049047] ... version: 2 [ 0.050017] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052020] ... value mask: 0000ffffffffffff [ 0.053017] ... max period: 00007fffffffffff [ 0.054017] ... fixed-purpose events: 3 [ 0.055020] ... event mask: 000000070000000f [ 0.057000] rcu: Hierarchical SRCU implementation. [ 0.059399] smp: Bringing up secondary CPUs ... [ 0.060742] x86: Booting SMP configuration: [ 0.061047] .... node #0, CPUs: #1 #2 #3 [ 0.069675] smp: Brought up 1 node, 4 CPUs [ 0.072026] smpboot: Max logical packages: 1 [ 0.073030] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152081] node 0 deferred pages initialised in 76ms [ 0.158187] devtmpfs: initialized [ 0.160367] x86/mm: Memory block size: 128MB [ 0.166225] gcov: version magic: 0x41383552 [ 0.171427] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.179213] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.182657] pinctrl core: initialized pinctrl subsystem [ 0.185917] [ 0.187012] ************************************************************* [ 0.189021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193032] ** ** [ 0.195020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.199020] ** ** [ 0.202016] ** This means that this kernel is built to expose internal ** [ 0.206024] ** IOMMU data structures, which may compromise security on ** [ 0.209017] ** your system. ** [ 0.214023] ** ** [ 0.217018] ** If you see this message and you are not debugging the ** [ 0.219021] ** kernel, report this immediately to your vendor! ** [ 0.222017] ** ** [ 0.224018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228022] ************************************************************* [ 0.232217] NET: Registered protocol family 16 [ 0.235574] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.239092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.247263] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.263137] cpuidle: using governor menu [ 0.265000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.277123] PCI: Using configuration type 1 for base access [ 0.298174] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.305785] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.306092] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.310291] cryptd: max_cpu_qlen set to 1000 [ 0.313277] ACPI: Added _OSI(Module Device) [ 0.314015] ACPI: Added _OSI(Processor Device) [ 0.315025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.317036] ACPI: Added _OSI(Processor Aggregator Device) [ 0.323553] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.331830] ACPI: Interpreter enabled [ 0.333147] ACPI: PM: (supports S0 S3 S4 S5) [ 0.334017] ACPI: Using IOAPIC for interrupt routing [ 0.336139] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.343000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.368929] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.371042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.373020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.376104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.380000] acpiphp: Slot [2] registered [ 0.380000] acpiphp: Slot [5] registered [ 0.382114] acpiphp: Slot [6] registered [ 0.383109] acpiphp: Slot [7] registered [ 0.385096] acpiphp: Slot [8] registered [ 0.386077] acpiphp: Slot [9] registered [ 0.387106] acpiphp: Slot [10] registered [ 0.388104] acpiphp: Slot [3] registered [ 0.390097] acpiphp: Slot [4] registered [ 0.391179] acpiphp: Slot [11] registered [ 0.392092] acpiphp: Slot [12] registered [ 0.393123] acpiphp: Slot [13] registered [ 0.394111] acpiphp: Slot [14] registered [ 0.396084] acpiphp: Slot [15] registered [ 0.397134] acpiphp: Slot [16] registered [ 0.398184] acpiphp: Slot [17] registered [ 0.400099] acpiphp: Slot [18] registered [ 0.401080] acpiphp: Slot [19] registered [ 0.402073] acpiphp: Slot [20] registered [ 0.403084] acpiphp: Slot [21] registered [ 0.405086] acpiphp: Slot [22] registered [ 0.406088] acpiphp: Slot [23] registered [ 0.407110] acpiphp: Slot [24] registered [ 0.409085] acpiphp: Slot [25] registered [ 0.410085] acpiphp: Slot [26] registered [ 0.411090] acpiphp: Slot [27] registered [ 0.412090] acpiphp: Slot [28] registered [ 0.414212] acpiphp: Slot [29] registered [ 0.416296] acpiphp: Slot [30] registered [ 0.419388] acpiphp: Slot [31] registered [ 0.420219] PCI host bridge to bus 0000:00 [ 0.422021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.426054] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.430037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.434027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.436033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.438021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.440303] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.442995] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.447423] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.457019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.462015] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.464019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.466019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.468017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.471077] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.475923] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.478052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.481993] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.487016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.498966] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.502996] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.510647] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.532021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.540059] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.548000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.553232] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.564030] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.573024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.593023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.609615] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.627000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.637021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.651022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.659801] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.663022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.671022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.693022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.706751] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.716027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.722082] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.741022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.747000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.754023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.760025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.777085] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.790178] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.792602] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.795583] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.798366] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.800349] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.805074] iommu: Default domain type: Passthrough [ 0.806825] SCSI subsystem initialized [ 0.808373] ACPI: bus type USB registered [ 0.810304] usbcore: registered new interface driver usbfs [ 0.813143] usbcore: registered new interface driver hub [ 0.815118] usbcore: registered new device driver usb [ 0.817346] pps_core: LinuxPPS API ver. 1 registered [ 0.820014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.823237] PTP clock support registered [ 0.826071] EDAC MC: Ver: 3.0.0 [ 0.827209] PCI: Using ACPI for IRQ routing [ 0.828977] NetLabel: Initializing [ 0.829000] NetLabel: domain hash size = 128 [ 0.829000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.831137] NetLabel: unlabeled traffic allowed by default [ 0.833534] vgaarb: loaded [ 0.835583] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.838023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.844704] clocksource: Switched to clocksource kvm-clock [ 1.039244] VFS: Disk quotas dquot_6.6.0 [ 1.041486] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.045227] *** VALIDATE ramfs *** [ 1.047210] *** VALIDATE hugetlbfs *** [ 1.049479] pnp: PnP ACPI init [ 1.053804] pnp: PnP ACPI: found 6 devices [ 1.072209] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.076082] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.078627] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.081072] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.083834] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.086662] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.089808] NET: Registered protocol family 2 [ 1.092669] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.098523] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.102974] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.108766] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.112532] TCP: Hash tables configured (established 65536 bind 65536) [ 1.115401] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.118476] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.121983] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.125918] NET: Registered protocol family 1 [ 1.129963] RPC: Registered named UNIX socket transport module. [ 1.132649] RPC: Registered udp transport module. [ 1.134614] RPC: Registered tcp transport module. [ 1.136512] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.138995] NET: Registered protocol family 44 [ 1.141171] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.143729] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.146148] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.148547] PCI: CLS 0 bytes, default 64 [ 1.150452] Unpacking initramfs... [ 2.948518] debug: unmapping init [mem 0xffff992f3cc54000-0xffff992f3ffbffff] [ 2.972470] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.989228] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.010621] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.790775] Initialise system trusted keyrings [ 3.793436] Key type blacklist registered [ 3.796500] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.813190] zbud: loaded [ 3.819307] *** VALIDATE nfs *** [ 3.821236] *** VALIDATE nfs4 *** [ 3.824809] pstore: using deflate compression [ 3.830159] Platform Keyring initialized [ 4.083182] NET: Registered protocol family 38 [ 4.086711] Key type asymmetric registered [ 4.090761] Asymmetric key parser 'x509' registered [ 4.094488] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.100053] io scheduler mq-deadline registered [ 4.102885] io scheduler kyber registered [ 4.105734] io scheduler bfq registered [ 4.108446] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.114622] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.119388] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.124377] ACPI: Power Button [PWRF] [ 4.135619] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.146065] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.167772] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.179415] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.209519] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.247781] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.278076] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.284394] Non-volatile memory driver v1.3 [ 4.285792] Linux agpgart interface v0.103 [ 4.322454] virtio_blk virtio1: [vda] 139280 512-byte logical blocks (71.3 MB/68.0 MiB) [ 4.325792] vda: detected capacity change from 0 to 71311360 [ 4.345397] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.348395] vdb: detected capacity change from 0 to 1073741824 [ 4.368764] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.372746] vdc: detected capacity change from 0 to 2621440000 [ 4.396490] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.400234] vdd: detected capacity change from 0 to 2621440000 [ 4.422235] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.426642] vde: detected capacity change from 0 to 4294967296 [ 4.445486] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.449215] vdf: detected capacity change from 0 to 4294967296 [ 4.456922] libphy: Fixed MDIO Bus: probed [ 4.464728] usbcore: registered new interface driver usbserial_generic [ 4.467964] usbserial: USB Serial support registered for generic [ 4.471127] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.477036] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.479210] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.482214] mousedev: PS/2 mouse device common for all mice [ 4.486390] rtc_cmos 00:05: RTC can wake from S4 [ 4.487789] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.494695] rtc_cmos 00:05: registered as rtc0 [ 4.502283] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.507132] intel_pstate: CPU model not supported [ 4.512347] hid: raw HID events driver (C) Jiri Kosina [ 4.514725] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.518164] usbcore: registered new interface driver usbhid [ 4.524972] usbhid: USB HID core driver [ 4.527412] drop_monitor: Initializing network drop monitor service [ 4.530370] Initializing XFRM netlink socket [ 4.532514] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.536305] NET: Registered protocol family 10 [ 4.547252] Segment Routing with IPv6 [ 4.549026] NET: Registered protocol family 17 [ 4.551715] mpls_gso: MPLS GSO support [ 4.558328] RAS: Correctable Errors collector initialized. [ 4.560948] AVX version of gcm_enc/dec engaged. [ 4.562633] AES CTR mode by8 optimization enabled [ 4.850944] sched_clock: Marking stable (4850765455, 0)->(6398809628, -1548044173) [ 4.859901] registered taskstats version 1 [ 4.864469] Loading compiled-in X.509 certificates [ 4.868728] zswap: loaded using pool lzo/zbud [ 4.936860] Key type big_key registered [ 4.954403] Key type encrypted registered [ 4.956348] ima: No TPM chip found, activating TPM-bypass! [ 4.958648] ima: Allocated hash algorithm: sha1 [ 4.960598] ima: No architecture policies found [ 4.962509] evm: Initialising EVM extended attributes: [ 4.964747] evm: security.selinux [ 4.966110] evm: security.ima [ 4.967330] evm: security.capability [ 4.968851] evm: HMAC attrs: 0x1 [ 4.971622] rtc_cmos 00:05: setting system clock to 2026-06-16 14:57:50 UTC (1781621870) [ 4.977862] debug: unmapping init [mem 0xffffffffbd803000-0xffffffffbd9fffff] [ 4.981230] debug: unmapping init [mem 0xffffffffbc582000-0xffffffffbc858fff] [ 4.991205] Write protecting the kernel read-only data: 28672k [ 4.994392] debug: unmapping init [mem 0xffffffffbac03000-0xffffffffbadfffff] [ 4.997468] debug: unmapping init [mem 0xffffffffbb514000-0xffffffffbb5fffff] [ 5.792336] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.812784] systemd[1]: Detected virtualization kvm. [ 5.815821] systemd[1]: Detected architecture x86-64. [ 5.819412] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.901400] systemd[1]: No hostname configured. [ 5.909132] systemd[1]: Set hostname to . [ 5.912765] random: systemd: uninitialized urandom read (16 bytes read) [ 5.916582] systemd[1]: Initializing machine ID from random generator. [ 6.385232] random: ln: uninitialized urandom read (6 bytes read) [ 6.980934] random: systemd: uninitialized urandom read (16 bytes read) [ 6.985413] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 6.996685] random: systemd: uninitialized urandom read (16 bytes read) [ 7.001461] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 7.014451] random: systemd: uninitialized urandom read (16 bytes read) [ 7.019497] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.612754] device-mapper: uevent: version 1.0.3 [ 8.615473] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ 10.433224] random: fast init done [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.441342] virtio_net virtio0 ens2: renamed from eth0 [ 13.961187] scsi host0: ata_piix [ 14.025635] scsi host1: ata_piix [ 14.029179] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.036685] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.532725] random: crng init done [ 16.535597] random: 5 urandom warning(s) missed due to ratelimiting [ 19.145576] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.250615] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.927502] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.443929] SELinux: Disabled at runtime. [ 24.500870] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.544694] systemd[1]: Detected virtualization kvm. [ 24.546794] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.689891] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.695846] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.720929] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.733702] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.747292] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.810748] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.818721] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ 27.508997] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 30.457995] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.482889] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.063080] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [* ] A start job is running for Configur…-only root support (9s / no limit)[ 35.973456] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 36.081938] EDAC sbridge: Ver: 1.1.2 [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root[ 39.367793] Key type dns_resolver registered support (13s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit)[ 40.409353] NFS: Registering the id_resolver key type [ 40.411592] Key type id_resolver registered [ 40.413560] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg208-server login: [ 61.039574] hrtimer: interrupt took 6964391 ns [ 106.080827] libcfs: loading out-of-tree module taints kernel. [ 106.131166] Key type ._llcrypt registered [ 106.133299] Key type .llcrypt registered [ 106.257884] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 119.628574] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 120.584364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 120.596131] alg: No test for adler32 (adler32-zlib) [ 122.067371] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 122.599218] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 124.311373] Key type lgssc registered [ 125.608537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 139.158334] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 168.928648] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 178.024117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 178.047097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.201829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 179.222899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 179.274434] Lustre: lustre-MDT0000: new disk, initializing [ 179.331944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.346866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 182.143615] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.033317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.128852] Lustre: 6511:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 193.160104] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 193.164571] Lustre: Skipped 1 previous similar message [ 193.275821] Lustre: lustre-MDT0001: new disk, initializing [ 193.379691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 193.402695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 193.414565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.139498] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.148688] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 208.135284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 208.309568] Lustre: lustre-OST0000: new disk, initializing [ 208.312554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 208.316505] Lustre: 8460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 208.375459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.107055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 209.118120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 209.209371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 212.964231] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.641667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.740330] Lustre: lustre-OST0001: new disk, initializing [ 225.743347] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 225.747750] Lustre: 9533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.803762] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.811644] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.451603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.460138] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.505789] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 239.727839] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 243.472076] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.874233] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.092066] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing yml_node [ 254.196851] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 255.984413] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 257.799921] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 259.088438] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Jun 16 11:02:02 EDT 2026 [ 271.380615] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 272.975398] Lustre: DEBUG MARKER: === replay-single: start setup 11:02:16 (1781622136) === [ 277.319563] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing check_config_client /mnt/lustre [ 292.015336] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 294.545962] Lustre: 13338:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 298.636310] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 303.865454] Lustre: DEBUG MARKER: === replay-single: finish setup 11:02:47 (1781622167) === [ 305.312504] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 11:02:49 (1781622169) [ 311.930970] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 313.977832] Lustre: Failing over lustre-MDT0000 [ 314.266478] Lustre: server umount lustre-MDT0000 complete [ 316.384074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 316.388265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.433117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.875247] LustreError: 8468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.889422] LustreError: 8468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 323.552433] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.565961] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 328.096728] LustreError: 11288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.154755] LustreError: 8468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.183324] LustreError: 8468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 334.839725] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781622183/real 1781622183] req@ffff992fc0d23100 x1868166057789184/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781622199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 334.989144] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.322993] LustreError: 8468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.360908] LustreError: 8468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 344.747660] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 344.755708] LDISKFS-fs (dm-0): recovery complete [ 344.797608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 348.479703] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.500625] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 360.782441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.231044] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 366.064511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 366.166122] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 366.954373] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 380.047898] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 383.168840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 396.442969] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 11:04:19 (1781622259) [ 399.087502] Lustre: Failing over lustre-OST0000 [ 399.170162] Lustre: server umount lustre-OST0000 complete [ 399.331624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 399.343774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.354641] Lustre: Skipped 2 previous similar messages [ 399.376149] LustreError: 8443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.407428] LustreError: 8443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 401.890174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.709321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 419.862116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.483412] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 421.856612] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 421.861914] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 421.883673] Lustre: Skipped 3 previous similar messages [ 426.400759] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 437.018638] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 439.228820] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 449.255881] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 11:05:12 (1781622312) [ 456.680893] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 458.792048] Lustre: Failing over lustre-MDT0000 [ 459.388720] Lustre: server umount lustre-MDT0000 complete [ 460.772982] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 460.810819] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 460.808687] Lustre: Skipped 2 previous similar messages [ 460.832429] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 477.169373] Lustre: 3650:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781622326/real 1781622326] req@ffff992fabc24380 x1868166057864448/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781622342 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 477.213173] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 482.857272] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 482.860231] LDISKFS-fs (dm-0): recovery complete [ 482.868482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.394116] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xee9cf87cdd3494cf [ 487.403242] Lustre: MGC192.168.202.108@tcp: Connection restored to 0@lo (at 0@lo) [ 487.413366] Lustre: Skipped 1 previous similar message [ 487.634310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.198227] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 493.056179] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 493.063487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 494.636300] Lustre: lustre-MDT0000: Denying connection for new client 513d9828-bd2a-4be4-b6f7-245e47fb71bf (at 192.168.202.8@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 500.027246] Lustre: lustre-MDT0000: Denying connection for new client 513d9828-bd2a-4be4-b6f7-245e47fb71bf (at 192.168.202.8@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 505.148208] Lustre: lustre-MDT0000: Denying connection for new client 513d9828-bd2a-4be4-b6f7-245e47fb71bf (at 192.168.202.8@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 510.274167] Lustre: lustre-MDT0000: Denying connection for new client 513d9828-bd2a-4be4-b6f7-245e47fb71bf (at 192.168.202.8@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 515.388059] Lustre: lustre-MDT0000: Denying connection for new client 513d9828-bd2a-4be4-b6f7-245e47fb71bf (at 192.168.202.8@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 525.636844] Lustre: lustre-MDT0000: Denying connection for new client 513d9828-bd2a-4be4-b6f7-245e47fb71bf (at 192.168.202.8@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:37 [ 525.655582] Lustre: Skipped 1 previous similar message [ 546.109248] Lustre: lustre-MDT0000: Denying connection for new client 513d9828-bd2a-4be4-b6f7-245e47fb71bf (at 192.168.202.8@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:17 [ 546.134325] Lustre: Skipped 3 previous similar messages [ 563.500599] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 563.507111] Lustre: 18243:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c3d4a5fd-dc6a-4751-b24e-381560589e66@ [ 563.519691] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 563.533309] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 563.533737] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 563.561056] Lustre: Skipped 2 previous similar messages [ 563.587578] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 563.607637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 576.414767] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 11:07:19 (1781622439) [ 584.201618] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 586.428182] Lustre: Failing over lustre-MDT0000 [ 586.725802] Lustre: server umount lustre-MDT0000 complete [ 589.280197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 589.301389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.330747] Lustre: Skipped 1 previous similar message [ 589.345304] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 589.363197] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 606.689305] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781622455/real 1781622455] req@ffff992fc0d3fb80 x1868166057924864/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781622471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 606.720611] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.990134] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 609.992189] LDISKFS-fs (dm-0): recovery complete [ 609.997772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.936755] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xee9cf87cdd3498e9 [ 616.949106] Lustre: MGC192.168.202.108@tcp: Connection restored to 0@lo (at 0@lo) [ 617.244319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 617.288822] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 622.283917] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 622.569359] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 624.935522] Lustre: lustre-MDT0000: Denying connection for new client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 624.944946] Lustre: Skipped 3 previous similar messages [ 691.514868] Lustre: lustre-MDT0000: Denying connection for new client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:00 [ 691.540243] Lustre: Skipped 12 previous similar messages [ 692.500187] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 692.505044] Lustre: 19973:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 513d9828-bd2a-4be4-b6f7-245e47fb71bf@ [ 692.515038] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 692.530157] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 692.531096] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 692.555619] Lustre: Skipped 3 previous similar messages [ 692.599049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 692.601822] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 708.324894] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 11:09:31 (1781622571) [ 716.925687] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 719.018879] Lustre: Failing over lustre-MDT0000 [ 719.197470] Lustre: server umount lustre-MDT0000 complete [ 719.843727] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.857975] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 719.870134] Lustre: Skipped 5 previous similar messages [ 719.899866] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 31 previous similar messages [ 741.346025] Lustre: 3649:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781622590/real 1781622590] req@ffff992f8558b100 x1868166057986816/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781622606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 741.375395] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.361720] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 742.364401] LDISKFS-fs (dm-0): recovery complete [ 742.375553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 750.633567] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xee9cf87cdd349d8f [ 750.655524] Lustre: MGC192.168.202.108@tcp: Connection restored to 0@lo (at 0@lo) [ 751.045375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.125738] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 752.684965] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 756.276366] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 756.332525] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 756.427553] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 756.428442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 766.912609] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 768.975607] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 777.986328] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 11:10:41 (1781622641) [ 786.353714] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 789.022268] Lustre: Failing over lustre-MDT0000 [ 789.382745] Lustre: server umount lustre-MDT0000 complete [ 792.032581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.055764] Lustre: Skipped 3 previous similar messages [ 808.419294] Lustre: 3648:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781622657/real 1781622657] req@ffff992fabd6fb80 x1868166058025216/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781622673 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 808.449597] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 814.059622] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 814.062633] LDISKFS-fs (dm-0): recovery complete [ 814.069266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 817.962877] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 819.454284] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 823.036788] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 823.278664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 823.290293] Lustre: Skipped 4 previous similar messages [ 823.422453] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 823.469292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 823.469440] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 833.081909] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 835.342183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 844.538335] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 11:11:48 (1781622708) [ 852.542298] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 854.605871] Lustre: Failing over lustre-MDT0000 [ 854.808691] Lustre: server umount lustre-MDT0000 complete [ 859.105843] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.121537] Lustre: Skipped 3 previous similar messages [ 875.487713] Lustre: 3650:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781622724/real 1781622724] req@ffff992e87f1b800 x1868166058062976/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781622740 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 875.526307] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 877.911134] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 877.913663] LDISKFS-fs (dm-0): recovery complete [ 877.934692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.962865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 885.973192] Lustre: Skipped 1 previous similar message [ 886.014282] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 887.005553] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 890.836256] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 891.505865] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 891.556952] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:193) [ 891.557118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 901.141032] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 903.027907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 911.814867] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 11:12:55 (1781622775) [ 919.325298] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 921.369881] Lustre: Failing over lustre-MDT0000 [ 921.730111] Lustre: server umount lustre-MDT0000 complete [ 938.464139] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781622787/real 1781622787] req@ffff992fbad57480 x1868166058098944/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781622803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 938.483618] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 944.795659] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 944.798566] LDISKFS-fs (dm-0): recovery complete [ 944.808558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 948.706125] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fc1fb7b80 x1868166058107520/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 949.161350] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 950.439716] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 954.051723] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 954.349414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 954.359701] Lustre: Skipped 7 previous similar messages [ 954.440881] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 954.468651] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 954.475993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 963.949412] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 965.712050] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 976.793434] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 11:13:59 (1781622839) [ 985.033269] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 987.093464] Lustre: Failing over lustre-MDT0000 [ 987.355792] Lustre: server umount lustre-MDT0000 complete [ 990.180117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.182953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 990.185860] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.185871] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 138 previous similar messages [ 990.213219] Lustre: Skipped 7 previous similar messages [ 1005.535375] Lustre: 3648:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781622855/real 1781622855] req@ffff992fc0ba8380 x1868166058137856/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781622871 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1005.565546] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1009.203420] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1009.208113] LDISKFS-fs (dm-0): recovery complete [ 1009.219657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.780143] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fbad55500 x1868166058146560/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1016.316145] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1017.161994] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1021.113850] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1021.461194] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1021.491125] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 1021.491591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1030.975060] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1033.079158] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1041.798112] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 11:15:05 (1781622905) [ 1042.989424] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1042.992786] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1042.997443] LustreError: 6519:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992fa63aed80 x1868166043163008/t38654705666(0) o35->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:344/0 lens 392/456 e 0 to 0 dl 1781622919 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1051.926139] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1054.447935] Lustre: Failing over lustre-MDT0000 [ 1055.060707] Lustre: server umount lustre-MDT0000 complete [ 1057.257655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1073.631136] Lustre: 3650:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781622922/real 1781622922] req@ffff992f9f2e9500 x1868166058177024/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781622938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1073.652504] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1077.178623] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1077.182754] LDISKFS-fs (dm-0): recovery complete [ 1077.194596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.872030] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fbcd03100 x1868166058185600/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1084.473735] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1089.488411] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1089.626043] Lustre: 6519:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fbcd00a80 x1868166043163008/t38654705666(0) o35->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:391/0 lens 392/456 e 0 to 0 dl 1781622966 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1089.665639] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1089.665728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1098.682307] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1100.091545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1108.130941] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 11:16:11 (1781622971) [ 1115.326292] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1117.455227] Lustre: Failing over lustre-MDT0000 [ 1117.812208] Lustre: server umount lustre-MDT0000 complete [ 1138.474584] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1138.477249] LDISKFS-fs (dm-0): recovery complete [ 1138.487298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1146.072477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.074831] Lustre: Skipped 3 previous similar messages [ 1146.119680] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1146.185627] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1146.197949] Lustre: Skipped 1 previous similar message [ 1150.255843] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1151.509976] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1151.528092] Lustre: Skipped 1 previous similar message [ 1151.568931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1151.572498] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1158.578480] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1160.127577] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1169.649774] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 11:17:13 (1781623033) [ 1178.136844] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1179.214377] Lustre: *** cfs_fail_loc=114, val=0*** [ 1182.306691] Lustre: Failing over lustre-MDT0000 [ 1182.682133] Lustre: server umount lustre-MDT0000 complete [ 1187.295550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1203.680106] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781623052/real 1781623052] req@ffff992fc0e02680 x1868166058251008/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781623068 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1203.712699] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1203.721949] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1203.745441] LustreError: Skipped 1 previous similar message [ 1205.181374] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1205.188985] LDISKFS-fs (dm-0): recovery complete [ 1205.202791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.317446] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1219.569172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1219.583421] Lustre: Skipped 15 previous similar messages [ 1219.597774] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1219.718695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1219.732624] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1228.918239] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1230.749378] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1238.883898] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 11:18:22 (1781623102) [ 1246.305689] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1247.178966] Lustre: *** cfs_fail_loc=128, val=0*** [ 1249.637729] Lustre: Failing over lustre-MDT0000 [ 1249.849242] Lustre: server umount lustre-MDT0000 complete [ 1250.281633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.284473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1250.303247] Lustre: Skipped 15 previous similar messages [ 1271.315942] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1271.319192] LDISKFS-fs (dm-0): recovery complete [ 1271.327682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.879093] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xee9cf87cdd34c61c [ 1279.848458] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1281.595097] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1281.598471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1288.322302] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1289.868988] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1298.084555] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 11:19:21 (1781623161) [ 1305.684523] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1308.055807] Lustre: Failing over lustre-MDT0000 [ 1310.298254] Lustre: server umount lustre-MDT0000 complete [ 1333.777421] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1333.781276] LDISKFS-fs (dm-0): recovery complete [ 1333.787922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.847721] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fc0a9a300 x1868166058329856/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1342.405850] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1344.693733] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1344.703728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1350.868931] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1352.298930] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1360.175645] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 11:20:23 (1781623223) [ 1366.745885] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1368.826864] Lustre: Failing over lustre-MDT0000 [ 1369.188354] Lustre: server umount lustre-MDT0000 complete [ 1390.461306] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1390.464960] LDISKFS-fs (dm-0): recovery complete [ 1390.483765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.064595] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1397.068927] Lustre: Skipped 2 previous similar messages [ 1401.566298] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1402.468453] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1402.471087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1410.675486] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1412.430886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1420.434656] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 11:21:23 (1781623283) [ 1426.680740] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1433.755552] Lustre: Failing over lustre-MDT0000 [ 1433.922295] Lustre: server umount lustre-MDT0000 complete [ 1438.176650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1453.458339] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1453.463056] LDISKFS-fs (dm-0): recovery complete [ 1453.484053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1455.430081] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1455.437298] Lustre: Skipped 4 previous similar messages [ 1457.009231] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1461.680404] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 1461.684772] Lustre: Skipped 4 previous similar messages [ 1461.729222] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1461.735016] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1466.091131] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1467.764558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1488.084435] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 11:22:31 (1781623351) [ 1494.519350] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1496.423667] Lustre: Failing over lustre-MDT0000 [ 1496.710984] Lustre: server umount lustre-MDT0000 complete [ 1504.061168] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1504.073373] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 252 previous similar messages [ 1515.488968] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781623365/real 1781623365] req@ffff992fbcc77480 x1868166058489344/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781623381 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1515.535213] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1515.550399] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1515.565494] LustreError: Skipped 4 previous similar messages [ 1518.163766] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1518.165651] LDISKFS-fs (dm-0): recovery complete [ 1518.172483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1529.948627] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1531.462942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1531.462942] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1537.733790] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1539.295632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1548.361614] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 11:23:32 (1781623412) [ 1554.492360] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1556.053465] Lustre: Failing over lustre-MDT0000 [ 1556.315668] Lustre: server umount lustre-MDT0000 complete [ 1556.959977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1556.967314] LustreError: Skipped 1 previous similar message [ 1575.899325] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1575.902204] LDISKFS-fs (dm-0): recovery complete [ 1575.909346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.695576] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1581.609256] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1581.619744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1588.253510] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1589.451593] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1595.885138] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 11:24:19 (1781623459) [ 1601.816971] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1603.800325] Lustre: Failing over lustre-MDT0000 [ 1604.044501] Lustre: server umount lustre-MDT0000 complete [ 1624.043437] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1624.046348] LDISKFS-fs (dm-0): recovery complete [ 1624.061686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.777635] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xee9cf87cdd357db2 [ 1637.936411] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1639.475164] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1639.477897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1646.417239] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1647.975781] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1655.843541] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 11:25:19 (1781623519) [ 1663.049752] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1664.789181] Lustre: Failing over lustre-MDT0000 [ 1665.032960] Lustre: server umount lustre-MDT0000 complete [ 1685.293796] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1685.296892] LDISKFS-fs (dm-0): recovery complete [ 1685.312518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.473225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1685.540598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.544283] Lustre: Skipped 8 previous similar messages [ 1685.567535] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1685.576719] Lustre: Skipped 4 previous similar messages [ 1688.895625] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1690.734318] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1690.739315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1696.796918] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1698.390123] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1706.511587] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 11:26:10 (1781623570) [ 1713.931992] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1715.772399] Lustre: Failing over lustre-MDT0000 [ 1716.089439] Lustre: server umount lustre-MDT0000 complete [ 1716.195321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1716.205727] LustreError: Skipped 1 previous similar message [ 1737.569705] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1737.571755] LDISKFS-fs (dm-0): recovery complete [ 1737.584406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.793040] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xee9cf87cdd35885c [ 1741.809270] Lustre: MGC192.168.202.108@tcp: Connection restored to 0@lo (at 0@lo) [ 1741.831160] Lustre: Skipped 37 previous similar messages [ 1746.433811] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1747.519898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1747.519941] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1754.451807] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1755.558285] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1761.690882] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 11:27:05 (1781623625) [ 1767.687600] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1769.547054] Lustre: Failing over lustre-MDT0000 [ 1769.822699] Lustre: server umount lustre-MDT0000 complete [ 1773.026181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.049433] Lustre: Skipped 32 previous similar messages [ 1789.745149] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1789.747785] LDISKFS-fs (dm-0): recovery complete [ 1789.758902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.625154] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992e86683480 x1868166058663168/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1802.151705] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1804.332707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1804.336404] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1809.114774] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1810.391644] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1817.423674] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 11:28:00 (1781623680) [ 1825.810457] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1827.453346] Lustre: Failing over lustre-MDT0000 [ 1827.613651] Lustre: server umount lustre-MDT0000 complete [ 1847.475425] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1847.478153] LDISKFS-fs (dm-0): recovery complete [ 1847.487476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.481868] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992e88e53800 x1868166058695168/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1860.406315] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1862.302188] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1862.310929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1869.127850] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1870.180420] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1876.952357] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 11:29:00 (1781623740) [ 1883.646208] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1885.311835] Lustre: Failing over lustre-MDT0000 [ 1887.478144] Lustre: server umount lustre-MDT0000 complete [ 1909.082823] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1909.085502] LDISKFS-fs (dm-0): recovery complete [ 1909.100177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1913.824348] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fbcb94700 x1868166058728064/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1917.991489] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1919.566466] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1919.568380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1925.739554] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1927.116121] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1934.496512] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 11:29:58 (1781623798) [ 1941.210387] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1943.299878] Lustre: Failing over lustre-MDT0000 [ 1943.634491] Lustre: server umount lustre-MDT0000 complete [ 1964.116472] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1964.119882] LDISKFS-fs (dm-0): recovery complete [ 1964.131843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.175319] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xee9cf87cdd359ed6 [ 1971.519039] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1971.526779] Lustre: Skipped 8 previous similar messages [ 1974.813603] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1976.844817] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1976.854505] Lustre: Skipped 8 previous similar messages [ 1976.883371] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 1976.892426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 1982.318871] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1983.726406] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1991.703972] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 11:30:55 (1781623855) [ 1999.069866] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2000.897099] Lustre: Failing over lustre-MDT0000 [ 2001.239185] Lustre: server umount lustre-MDT0000 complete [ 2002.407298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2002.415476] LustreError: Skipped 3 previous similar messages [ 2022.857604] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2022.860040] LDISKFS-fs (dm-0): recovery complete [ 2022.869179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.837578] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2033.721471] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 2033.727839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:897) [ 2039.961285] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2041.637559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2048.831783] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 11:31:52 (1781623912) [ 2054.337126] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2055.856328] Lustre: Failing over lustre-MDT0000 [ 2056.147098] Lustre: server umount lustre-MDT0000 complete [ 2074.592234] Lustre: 3650:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781623924/real 1781623924] req@ffff992fbcc4b480 x1868166058816256/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781623940 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2074.630545] Lustre: 3650:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 2074.638139] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2074.649363] LustreError: Skipped 9 previous similar messages [ 2076.184421] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2076.187384] LDISKFS-fs (dm-0): recovery complete [ 2076.194426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.832947] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fb6e65180 x1868166058823552/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2089.121590] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2090.570314] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2090.570986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2096.283987] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2097.406639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2104.230529] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 11:32:48 (1781623968) [ 2109.624574] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2111.312700] Lustre: Failing over lustre-MDT0000 [ 2111.506899] Lustre: server umount lustre-MDT0000 complete [ 2115.910681] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.926191] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 318 previous similar messages [ 2130.901391] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2130.903528] LDISKFS-fs (dm-0): recovery complete [ 2130.908986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.072354] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (not set up) [ 2131.087831] Lustre: Skipped 3 previous similar messages [ 2134.885643] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2136.615106] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 2136.616084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 2142.208763] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2143.765657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2150.851486] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 11:33:34 (1781624014) [ 2156.301401] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2157.818608] Lustre: Failing over lustre-MDT0000 [ 2157.892713] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 2157.994865] Lustre: server umount lustre-MDT0000 complete [ 2177.321219] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2177.323785] LDISKFS-fs (dm-0): recovery complete [ 2177.329475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.571173] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2182.704669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 2182.706972] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:993) [ 2187.636478] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2188.700790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2195.200671] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 11:34:19 (1781624059) [ 2200.676743] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2202.452053] Lustre: Failing over lustre-MDT0000 [ 2202.653663] Lustre: server umount lustre-MDT0000 complete [ 2222.585890] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2222.588031] LDISKFS-fs (dm-0): recovery complete [ 2222.593813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.061099] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2230.071028] Lustre: Skipped 9 previous similar messages [ 2233.957667] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2235.506356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2235.508415] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2242.501640] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2243.882326] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2251.626976] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 11:35:15 (1781624115) [ 2258.590647] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2260.258354] Lustre: Failing over lustre-MDT0000 [ 2260.449340] Lustre: server umount lustre-MDT0000 complete [ 2280.895730] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2280.898063] LDISKFS-fs (dm-0): recovery complete [ 2280.904152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.878876] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2286.676895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2286.677654] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2293.752858] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2295.303817] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2302.396318] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:36:06 (1781624166) [ 2308.661599] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2310.254402] Lustre: Failing over lustre-MDT0000 [ 2310.432557] Lustre: server umount lustre-MDT0000 complete [ 2329.695797] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2329.699199] LDISKFS-fs (dm-0): recovery complete [ 2329.709130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.759802] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fbcf70700 x1868166058973824/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2338.066196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.070097] Lustre: Skipped 11 previous similar messages [ 2342.347626] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2343.407521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2343.423714] Lustre: Skipped 45 previous similar messages [ 2343.518143] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 2343.522390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 2350.833792] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2352.218509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2359.308421] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 11:37:03 (1781624223) [ 2364.143926] Lustre: 75313:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 13fc764b-ac08-40d4-bea6-629cf6c12fe2 at adminstrative request [ 2368.639368] Lustre: Failing over lustre-MDT0000 [ 2368.920897] Lustre: server umount lustre-MDT0000 complete [ 2386.577765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.171282] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2392.080754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2392.081279] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 2398.052595] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2399.308910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2403.583451] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2410.647325] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2415.764917] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 11:37:59 (1781624279) [ 2416.606911] Lustre: 77338:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 13fc764b-ac08-40d4-bea6-629cf6c12fe2 at adminstrative request [ 2423.154961] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:38:06 (1781624286) [ 2429.195627] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2430.832388] Lustre: Failing over lustre-MDT0000 [ 2431.241823] Lustre: server umount lustre-MDT0000 complete [ 2432.991915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2433.000527] Lustre: Skipped 47 previous similar messages [ 2450.859263] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2450.861213] LDISKFS-fs (dm-0): recovery complete [ 2450.866044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.426188] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2464.385343] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1124 to 0x2c0000401:1153) [ 2464.396166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1123 to 0x280000401:1153) [ 2469.923650] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2471.061375] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2477.249385] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:39:01 (1781624341) [ 2482.615275] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2484.256740] Lustre: Failing over lustre-MDT0000 [ 2484.446981] Lustre: server umount lustre-MDT0000 complete [ 2503.790178] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2503.793261] LDISKFS-fs (dm-0): recovery complete [ 2503.805304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.022808] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2509.361206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2509.361808] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2514.533273] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2515.762840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2523.615068] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:39:46 (1781624386) [ 2530.310783] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2532.016846] Lustre: Failing over lustre-MDT0000 [ 2532.239734] Lustre: server umount lustre-MDT0000 complete [ 2534.880085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2534.889441] LustreError: Skipped 7 previous similar messages [ 2551.750227] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2551.753700] LDISKFS-fs (dm-0): recovery complete [ 2551.762519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.672507] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2566.212408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2566.216440] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2571.730930] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2573.146710] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2579.940631] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 11:40:43 (1781624443) [ 2585.908797] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2587.255455] Lustre: Failing over lustre-MDT0000 [ 2587.490306] Lustre: server umount lustre-MDT0000 complete [ 2605.660775] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2605.662494] LDISKFS-fs (dm-0): recovery complete [ 2605.668052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.470843] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2608.957498] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2608.965218] Lustre: Skipped 11 previous similar messages [ 2611.192190] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2611.196499] Lustre: Skipped 11 previous similar messages [ 2611.228282] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2611.233379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2614.433282] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2615.299255] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2620.670596] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:41:24 (1781624484) [ 2625.844588] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2626.950050] Lustre: Failing over lustre-MDT0000 [ 2627.123429] Lustre: server umount lustre-MDT0000 complete [ 2646.551580] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2646.554221] LDISKFS-fs (dm-0): recovery complete [ 2646.562195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.383513] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2652.204362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 2652.207783] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 2657.882255] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2659.157383] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2665.428529] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:42:09 (1781624529) [ 2670.119173] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2671.467876] Lustre: Failing over lustre-MDT0000 [ 2671.686533] Lustre: server umount lustre-MDT0000 complete [ 2688.991218] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781624538/real 1781624538] req@ffff992fa6608380 x1868166059182080/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781624554 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2689.010167] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 2689.015367] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2689.021196] LustreError: Skipped 11 previous similar messages [ 2689.389442] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2689.391229] LDISKFS-fs (dm-0): recovery complete [ 2689.400331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.235377] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xee9cf87cdd35ed11 [ 2702.309190] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2704.977354] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 2704.980459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 2708.685195] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2709.855608] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2715.826820] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:42:59 (1781624579) [ 2721.454662] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2722.918630] Lustre: Failing over lustre-MDT0000 [ 2723.114538] Lustre: server umount lustre-MDT0000 complete [ 2725.345038] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.360761] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 292 previous similar messages [ 2741.224859] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2741.226449] LDISKFS-fs (dm-0): recovery complete [ 2741.230335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2750.943495] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fb6e64a80 x1868166059219584/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2754.693184] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2756.661843] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 2756.664084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 2762.657956] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2763.960513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2770.477624] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:43:54 (1781624634) [ 2775.618659] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2776.905624] Lustre: Failing over lustre-MDT0000 [ 2777.059049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2777.066979] Lustre: Skipped 3 previous similar messages [ 2779.135441] Lustre: server umount lustre-MDT0000 complete [ 2796.648873] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2796.651344] LDISKFS-fs (dm-0): recovery complete [ 2796.658212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.495639] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2802.202970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 2802.206501] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 2805.582332] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2806.883912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2812.628501] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:44:36 (1781624676) [ 2817.580411] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2818.760079] Lustre: Failing over lustre-MDT0000 [ 2818.900773] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 2818.958188] Lustre: server umount lustre-MDT0000 complete [ 2836.606943] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2836.609668] LDISKFS-fs (dm-0): recovery complete [ 2836.617705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.832926] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2836.840199] Lustre: Skipped 11 previous similar messages [ 2839.336941] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2842.190239] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 2842.192958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 2845.988628] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2847.051509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2853.099303] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:45:17 (1781624717) [ 2858.355257] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2859.507869] Lustre: Failing over lustre-MDT0000 [ 2859.667213] Lustre: server umount lustre-MDT0000 complete [ 2876.984699] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2876.986886] LDISKFS-fs (dm-0): recovery complete [ 2876.991600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.420415] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2882.562139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 2882.562413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 2886.016637] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2887.001684] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2892.163512] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 11:45:56 (1781624756) [ 2896.588296] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2897.645143] Lustre: Failing over lustre-MDT0000 [ 2897.857116] Lustre: server umount lustre-MDT0000 complete [ 2915.640420] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2915.642515] LDISKFS-fs (dm-0): recovery complete [ 2915.649459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2918.884070] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2920.979028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 2920.979236] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 2925.087128] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2926.082587] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2931.980194] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 11:46:35 (1781624795) [ 2932.706377] Lustre: 98884:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 13fc764b-ac08-40d4-bea6-629cf6c12fe2 at adminstrative request [ 2940.529918] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 11:46:44 (1781624804) [ 2944.527935] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2945.989543] Lustre: Failing over lustre-MDT0000 [ 2946.211651] Lustre: server umount lustre-MDT0000 complete [ 2954.865923] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2954.868145] LDISKFS-fs (dm-0): recovery complete [ 2954.872966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.078745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2955.082342] Lustre: Skipped 12 previous similar messages [ 2955.118429] Lustre: lustre-MDT0000: Aborting client recovery [ 2955.121407] LustreError: 100166:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2955.126016] Lustre: 100199:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2955.134948] Lustre: 100199:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 13fc764b-ac08-40d4-bea6-629cf6c12fe2@ [ 2955.143762] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2955.149169] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2955.160298] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2955.198134] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 2955.208418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1505) [ 2957.761859] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2960.354074] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2960.378326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2960.389145] Lustre: Skipped 53 previous similar messages [ 2973.922196] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 11:47:17 (1781624837) [ 2979.346337] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2980.490423] Lustre: Failing over lustre-MDT0000 [ 2980.694144] Lustre: server umount lustre-MDT0000 complete [ 2992.602750] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2992.606078] LDISKFS-fs (dm-0): recovery complete [ 2992.617805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.983684] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2993.005808] Lustre: lustre-MDT0000: Aborting client recovery [ 2993.008575] LustreError: 102028:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2993.013683] Lustre: 102061:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2993.019116] Lustre: 102061:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages [ 2993.035891] Lustre: 102061:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2993.041808] Lustre: 102061:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2993.045785] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2993.050380] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 2993.060705] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 2993.108851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 2993.123647] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1512 to 0x2c0000401:1537) [ 2998.263691] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3000.044124] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3015.789344] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3015.791067] Lustre: Skipped 1 previous similar message [ 3025.983497] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 11:48:08 (1781624888) [ 3037.926484] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3040.514837] Lustre: Failing over lustre-MDT0000 [ 3041.596091] Lustre: server umount lustre-MDT0000 complete [ 3044.327781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.344568] Lustre: Skipped 51 previous similar messages [ 3057.611972] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3057.619851] LDISKFS-fs (dm-0): recovery complete [ 3057.641252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.174363] Lustre: lustre-MDT0000: Aborting client recovery [ 3058.179918] LustreError: 103898:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3058.189706] Lustre: 103931:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3058.201201] Lustre: 103931:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages [ 3058.209743] Lustre: 103931:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3058.227874] Lustre: 103931:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3058.237587] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3058.263209] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3058.300568] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3058.405910] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 3058.406687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 3063.264728] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3063.459623] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3082.725224] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 11:49:06 (1781624946) [ 3083.770135] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3083.775337] LustreError: 6518:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992fa63b1880 x1868166044201856/t201863462916(0) o36->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:120/0 lens 512/456 e 0 to 0 dl 1781624960 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3087.225929] Lustre: Failing over lustre-MDT0000 [ 3087.435985] Lustre: server umount lustre-MDT0000 complete [ 3095.604857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.965629] Lustre: lustre-MDT0000: Aborting client recovery [ 3095.967699] LustreError: 105231:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3095.971510] Lustre: 105264:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3095.982486] Lustre: 105264:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages [ 3095.988124] Lustre: 105264:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3095.995138] Lustre: 105264:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3096.000548] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3096.007209] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3096.018220] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3096.062141] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 3096.072687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1601) [ 3100.187876] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3101.184138] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3120.226243] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3121.770671] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 11:49:45 (1781624985) [ 3128.358360] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3130.846762] Lustre: Failing over lustre-MDT0000 [ 3130.975694] Lustre: server umount lustre-MDT0000 complete [ 3143.091097] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3143.095414] LDISKFS-fs (dm-0): recovery complete [ 3143.110306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3143.622921] Lustre: lustre-MDT0000: Aborting client recovery [ 3143.628717] LustreError: 107195:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3143.634856] Lustre: 107229:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3143.641586] Lustre: 107229:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages [ 3143.646473] Lustre: 107229:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3143.653244] Lustre: 107229:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3143.659706] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3143.669222] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3143.689605] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3143.799042] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 3143.821443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1633) [ 3148.538087] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3148.785369] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3168.894315] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:50:32 (1781625032) [ 3201.487770] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3203.321872] Lustre: Failing over lustre-MDT0000 [ 3203.395697] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 3203.538323] Lustre: server umount lustre-MDT0000 complete [ 3205.088856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3205.092824] LustreError: Skipped 10 previous similar messages [ 3224.641384] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3224.643972] LDISKFS-fs (dm-0): recovery complete [ 3224.650623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3230.691892] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fabf99500 x1868166059585280/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3232.358461] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3232.361983] Lustre: Skipped 7 previous similar messages [ 3235.953300] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3236.451034] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3236.462791] Lustre: Skipped 7 previous similar messages [ 3236.507197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3236.507888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3244.981170] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3246.661031] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3266.340204] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:52:09 (1781625129) [ 3288.859260] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3299.114968] Lustre: Failing over lustre-MDT0000 [ 3299.469479] Lustre: server umount lustre-MDT0000 complete [ 3319.263125] Lustre: 3649:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781625168/real 1781625168] req@ffff992e8dc73480 x1868166059760640/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781625184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3319.285563] Lustre: 3649:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3319.293349] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3319.307356] LustreError: Skipped 11 previous similar messages [ 3319.955827] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3319.957534] LDISKFS-fs (dm-0): recovery complete [ 3319.964857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.483203] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3328.491344] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xee9cf87cdd387b2f [ 3328.505044] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 223 previous similar messages [ 3333.425422] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3337.323938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3337.323948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3343.341661] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3345.471967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3365.203976] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 11:53:48 (1781625228) [ 3367.399783] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3368.450728] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3368.463527] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3375.221159] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 11:53:58 (1781625238) [ 3399.982602] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3409.792586] Lustre: Failing over lustre-OST0000 [ 3409.885196] Lustre: server umount lustre-OST0000 complete [ 3429.331369] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3429.333735] LDISKFS-fs (dm-2): recovery complete [ 3429.347952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3434.036757] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3488.685963] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 11:55:52 (1781625352) [ 3495.638592] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3498.252462] Lustre: Failing over lustre-MDT0000 [ 3498.333016] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 3498.635502] Lustre: server umount lustre-MDT0000 complete [ 3520.132730] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3520.139403] LDISKFS-fs (dm-0): recovery complete [ 3520.153190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.507737] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3528.511528] Lustre: Skipped 20 previous similar messages [ 3532.608622] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3533.937242] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3533.939764] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3540.849560] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3542.291847] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3550.179564] LustreError: 116647:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3550.181220] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3551.200570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3559.773601] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 11:57:03 (1781625423) [ 3563.317206] LustreError: 66049:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3568.607391] LustreError: 66049:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3568.614979] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnecting [ 3568.666861] LustreError: 17145:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 waking [ 3570.217735] LustreError: 6518:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3575.263464] LustreError: 6518:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3575.271506] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnecting [ 3577.081634] LustreError: 6518:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3582.431189] LustreError: 6518:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3582.438788] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnecting [ 3583.908403] LustreError: 6517:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3589.087478] LustreError: 6517:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3590.652713] LustreError: 6518:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3595.744078] LustreError: 6518:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3595.751701] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnecting [ 3595.755371] Lustre: Skipped 1 previous similar message [ 3603.680126] LustreError: 66049:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3603.685298] LustreError: 66049:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3609.055806] LustreError: 66049:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3609.063333] LustreError: 66049:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3615.711241] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnecting [ 3615.727723] Lustre: Skipped 2 previous similar messages [ 3624.286445] LustreError: 66049:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3624.293964] LustreError: 66049:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3629.535164] LustreError: 66049:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3629.542614] LustreError: 66049:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3636.465265] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 11:58:20 (1781625500) [ 3638.068612] LustreError: 6515:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3648.315369] Lustre: lustre-MDT0000: Export ffff992f8b30f000 already connecting from 192.168.202.8@tcp [ 3653.442968] Lustre: lustre-MDT0000: Export ffff992f8b30f000 already connecting from 192.168.202.8@tcp [ 3658.555157] Lustre: lustre-MDT0000: Export ffff992f8b30f000 already connecting from 192.168.202.8@tcp [ 3663.683352] Lustre: lustre-MDT0000: Export ffff992f8b30f000 already connecting from 192.168.202.8@tcp [ 3663.692680] Lustre: Skipped 1 previous similar message [ 3668.795376] Lustre: lustre-MDT0000: Export ffff992f8b30f000 already connecting from 192.168.202.8@tcp [ 3678.127110] LustreError: 6515:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3678.136022] Lustre: 6515:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff992fbcf73800 x1868166046863744/t0(0) o38->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:0/0 lens 520/416 e 0 to 0 dl 1781625523 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3679.034429] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnecting [ 3679.049035] Lustre: Skipped 3 previous similar messages [ 3679.054557] LustreError: 6515:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3703.615374] Lustre: lustre-MDT0000: Export ffff992f8b30f000 already connecting from 192.168.202.8@tcp [ 3703.620283] Lustre: Skipped 1 previous similar message [ 3719.079243] LustreError: 6515:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3719.086347] Lustre: 6515:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff992fbcc48e00 x1868166046868224/t0(0) o38->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:0/0 lens 520/416 e 0 to 0 dl 1781625564 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3724.090085] LustreError: 6517:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3749.698819] Lustre: lustre-MDT0000: Export ffff992f8b30f000 already connecting from 192.168.202.8@tcp [ 3749.703032] Lustre: Skipped 4 previous similar messages [ 3764.196354] LustreError: 6517:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3764.206094] Lustre: 6517:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff992fbcff9180 x1868166046871808/t0(0) o38->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:0/0 lens 520/416 e 0 to 0 dl 1781625609 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3765.056185] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnecting [ 3765.065655] Lustre: Skipped 1 previous similar message [ 3765.070329] LustreError: 6518:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3789.628579] Lustre: lustre-MDT0000: Export ffff992f8b30f000 already connecting from 192.168.202.8@tcp [ 3789.637124] Lustre: Skipped 2 previous similar messages [ 3805.161029] LustreError: 6518:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3805.168608] Lustre: 6518:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff992e89e91500 x1868166046874880/t0(0) o38->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:0/0 lens 520/416 e 0 to 0 dl 1781625650 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3810.109229] LustreError: 66049:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3850.145320] LustreError: 66049:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3850.152630] Lustre: 66049:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff992fbc833100 x1868166046878464/t0(0) o38->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:0/0 lens 520/416 e 0 to 0 dl 1781625695 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3851.071200] LustreError: 6518:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3857.239127] LustreError: 6518:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout interrupted [ 3860.698345] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 12:02:04 (1781625724) [ 3866.423302] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3869.448429] Lustre: Failing over lustre-MDT0000 [ 3869.762879] Lustre: server umount lustre-MDT0000 complete [ 3871.715064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3871.721923] LustreError: Skipped 1 previous similar message [ 3871.724756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.735589] Lustre: Skipped 27 previous similar messages [ 3878.809941] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3878.815200] LDISKFS-fs (dm-0): recovery complete [ 3878.827665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.045166] Lustre: *** cfs_fail_loc=712, val=0*** [ 3879.048812] LustreError: 8444:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff992fc1fb5f80 x1868166060287616/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 3879.102339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3879.105550] Lustre: Skipped 8 previous similar messages [ 3879.133475] Lustre: lustre-MDT0000: Aborting client recovery [ 3879.138549] LustreError: 121501:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3879.145724] Lustre: 121534:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3879.151067] Lustre: 121534:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages [ 3879.160341] Lustre: 121534:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3879.166231] Lustre: 121534:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3879.170487] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3879.176701] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 3879.187968] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3879.225718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 3879.228648] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 3881.930045] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3884.514048] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3884.523046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3884.535960] Lustre: Skipped 36 previous similar messages [ 3895.240318] Lustre: Failing over lustre-MDT0000 [ 3895.481769] Lustre: server umount lustre-MDT0000 complete [ 3911.246187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.508253] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3912.517288] Lustre: Skipped 3 previous similar messages [ 3914.410802] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3916.789906] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3916.800367] Lustre: Skipped 3 previous similar messages [ 3916.838196] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 3916.839132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 3922.380974] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3923.978836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3930.848281] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 12:03:14 (1781625794) [ 3930.970951] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnecting [ 3930.977152] Lustre: Skipped 2 previous similar messages [ 3937.129969] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:03:21 (1781625801) [ 3937.866521] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3937.872309] LustreError: 106015:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992e89e90700 x1868166046937600/t0(0) o700->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:219/0 lens 264/248 e 0 to 0 dl 1781625814 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 3956.059274] Lustre: Failing over lustre-MDT0000 [ 3956.306592] Lustre: server umount lustre-MDT0000 complete [ 3957.731724] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3957.740445] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 75 previous similar messages [ 3972.698495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3972.791457] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3972.795951] LustreError: Skipped 3 previous similar messages [ 3975.734890] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3978.255155] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 3978.255433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 3982.254194] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3983.333098] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3991.085496] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:04:14 (1781625854) [ 3992.802571] Lustre: Failing over lustre-OST0000 [ 3992.887467] Lustre: server umount lustre-OST0000 complete [ 4008.349186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4012.022228] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4017.962672] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4018.845780] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4086.692259] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:05:50 (1781625950) [ 4091.488228] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4093.137430] Lustre: Failing over lustre-MDT0000 [ 4093.278454] Lustre: server umount lustre-MDT0000 complete [ 4111.586968] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4111.588843] LDISKFS-fs (dm-0): recovery complete [ 4111.593908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4114.587396] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4117.254078] Lustre: *** cfs_fail_loc=216, val=0*** [ 4117.256072] LustreError: 127664:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 4117.259126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 4118.304958] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 4183.473921] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:07:27 (1781626047) [ 4193.664167] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:07:37 (1781626057) [ 4195.579478] Lustre: Failing over lustre-MDT0000 [ 4195.766243] Lustre: server umount lustre-MDT0000 complete [ 4211.358899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4211.598950] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4211.609066] Lustre: Skipped 7 previous similar messages [ 4214.071094] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4216.811996] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4216.815095] LustreError: 129245:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992fc0a2f480 x1868166047059200/t0(0) o101->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:540/0 lens 328/344 e 0 to 0 dl 1781626135 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4274.508276] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnected, waiting for 2 clients in recovery for 1:04 [ 4274.521198] Lustre: 129245:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4274.553480] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 4274.553579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 4278.041815] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4278.852912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4284.815897] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:09:08 (1781626148) [ 4286.655679] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4292.261212] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4293.502455] Lustre: Failing over lustre-MDT0000 [ 4293.604091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4293.607614] Lustre: Skipped 3 previous similar messages [ 4293.673486] Lustre: server umount lustre-MDT0000 complete [ 4312.319103] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4312.322738] LDISKFS-fs (dm-0): recovery complete [ 4312.328388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.548908] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4317.745265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3137) [ 4317.745701] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 4323.802262] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4325.117309] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4331.088444] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:09:55 (1781626195) [ 4331.956917] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4337.944580] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4339.088881] Lustre: Failing over lustre-MDT0000 [ 4339.251976] Lustre: server umount lustre-MDT0000 complete [ 4357.242265] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4357.244182] LDISKFS-fs (dm-0): recovery complete [ 4357.251405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4360.315267] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4362.786754] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 4362.787810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3169) [ 4366.806236] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4367.774338] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4373.469376] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:10:37 (1781626237) [ 4374.117543] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4380.167501] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4381.388245] Lustre: Failing over lustre-MDT0000 [ 4381.547521] Lustre: server umount lustre-MDT0000 complete [ 4399.055257] Lustre: 3649:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781626248/real 1781626248] req@ffff992fbcc05880 x1868166060590080/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781626264 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4399.079893] Lustre: 3649:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4399.281971] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4399.284213] LDISKFS-fs (dm-0): recovery complete [ 4399.290106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4409.312984] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fb6dec700 x1868166060599808/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4412.009462] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4414.995236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3201) [ 4414.995711] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 4421.983641] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:11:25 (1781626285) [ 4423.916496] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4423.918596] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4423.923662] LustreError: 6519:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992f85593c50 x1868166047115648/t257698037777(0) o35->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:705/0 lens 392/456 e 0 to 0 dl 1781626300 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4426.083889] Lustre: Failing over lustre-MDT0000 [ 4426.276946] Lustre: server umount lustre-MDT0000 complete [ 4442.576339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.234451] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4448.258170] Lustre: 39767:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992f850d4000 x1868166047115648/t257698037777(0) o35->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:729/0 lens 392/456 e 0 to 0 dl 1781626324 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4448.278789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 4448.279051] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3201) [ 4452.813651] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4453.912462] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4460.016641] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:12:03 (1781626323) [ 4460.771723] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4460.774302] LustreError: 6517:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992f92cc7480 x1868166047129088/t261993005072(0) o36->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:31/0 lens 504/448 e 0 to 0 dl 1781626381 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4467.939239] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4469.213748] Lustre: Failing over lustre-MDT0000 [ 4469.367722] Lustre: server umount lustre-MDT0000 complete [ 4473.824385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4473.832406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4473.832750] Lustre: Skipped 41 previous similar messages [ 4473.838370] LustreError: Skipped 5 previous similar messages [ 4486.945515] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4486.947460] LDISKFS-fs (dm-0): recovery complete [ 4486.955221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4487.136241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4487.144265] Lustre: Skipped 9 previous similar messages [ 4489.712605] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4492.263381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4492.267390] Lustre: Skipped 38 previous similar messages [ 4492.311293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 4492.315513] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3233) [ 4492.349508] Lustre: 6518:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fbd187480 x1868166047129088/t261993005072(0) o36->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:62/0 lens 504/2880 e 0 to 0 dl 1781626412 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4496.483984] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4497.590442] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4503.263973] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:12:47 (1781626367) [ 4504.024784] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4504.028989] LustreError: 6518:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992fbce73480 x1868166047144576/t266287972368(0) o36->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:74/0 lens 504/448 e 0 to 0 dl 1781626424 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4505.588268] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4510.894941] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4512.167697] Lustre: Failing over lustre-MDT0000 [ 4512.367521] Lustre: server umount lustre-MDT0000 complete [ 4530.289265] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4530.291612] LDISKFS-fs (dm-0): recovery complete [ 4530.299156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.272501] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4534.010944] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4534.022225] Lustre: Skipped 9 previous similar messages [ 4535.798799] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4535.803343] Lustre: Skipped 9 previous similar messages [ 4535.833702] Lustre: 6515:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992f8558aa00 x1868166047144576/t266287972368(0) o36->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:106/0 lens 504/2880 e 0 to 0 dl 1781626456 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4535.843625] Lustre: 6515:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4535.849474] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 4535.849772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3297) [ 4542.991787] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:13:26 (1781626406) [ 4543.690420] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4543.692461] Lustre: Skipped 1 previous similar message [ 4543.696049] LustreError: 6515:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992fabec2300 x1868166047158272/t270582939664(0) o36->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:114/0 lens 504/448 e 0 to 0 dl 1781626464 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4543.717396] LustreError: 6515:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 1 previous similar message [ 4545.237689] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4545.240352] Lustre: Skipped 1 previous similar message [ 4550.937681] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4552.187814] Lustre: Failing over lustre-MDT0000 [ 4552.375930] Lustre: server umount lustre-MDT0000 complete [ 4559.675205] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4559.683624] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 175 previous similar messages [ 4569.909403] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4569.912783] LDISKFS-fs (dm-0): recovery complete [ 4569.923261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4573.063608] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4575.249697] Lustre: 6515:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fbcb94700 x1868166047158272/t270582939664(0) o36->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:145/0 lens 504/2880 e 0 to 0 dl 1781626495 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4575.253880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3329) [ 4575.259781] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3297) [ 4581.605722] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:14:05 (1781626445) [ 4582.434830] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4583.982280] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4583.984158] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4583.986570] LustreError: 6520:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992fabf04e00 x1868166047171968/t274877906960(0) o35->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:110/0 lens 392/456 e 0 to 0 dl 1781626460 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4589.772332] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4590.921521] Lustre: Failing over lustre-MDT0000 [ 4591.092499] Lustre: server umount lustre-MDT0000 complete [ 4608.601228] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4608.603243] LDISKFS-fs (dm-0): recovery complete [ 4608.610218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.698415] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4608.702726] LustreError: Skipped 9 previous similar messages [ 4611.536635] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4614.149667] Lustre: 39767:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fb6e66a00 x1868166047171968/t274877906960(0) o35->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:140/0 lens 392/456 e 0 to 0 dl 1781626490 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4614.156225] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3329) [ 4614.157096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 4620.824517] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:14:44 (1781626484) [ 4621.436869] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4621.439854] LustreError: 6517:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992f8619f100 x1868166047183360/t279172874255(0) o101->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:191/0 lens 664/608 e 0 to 0 dl 1781626541 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4682.043712] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnecting [ 4682.048049] Lustre: Skipped 4 previous similar messages [ 4682.052344] Lustre: 6515:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fbcc04e00 x1868166047183360/t279172874255(0) o101->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:252/0 lens 664/3488 e 0 to 0 dl 1781626602 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4685.678184] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:15:49 (1781626549) [ 4689.710558] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4690.752883] Lustre: Failing over lustre-MDT0000 [ 4690.936357] Lustre: server umount lustre-MDT0000 complete [ 4708.530887] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4708.533549] LDISKFS-fs (dm-0): recovery complete [ 4708.543309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4711.031777] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4714.017460] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3361) [ 4714.020311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 4717.224578] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4718.284665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4734.301660] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:16:38 (1781626598) [ 4739.323746] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4740.394684] Lustre: Failing over lustre-MDT0000 [ 4740.579583] Lustre: server umount lustre-MDT0000 complete [ 4757.830271] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4757.832548] LDISKFS-fs (dm-0): recovery complete [ 4757.837866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4760.344507] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4763.176581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 4763.181236] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3393) [ 4766.471193] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4767.395188] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4771.158835] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4778.748141] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:17:22 (1781626642) [ 4805.180271] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4806.238614] Lustre: Failing over lustre-MDT0000 [ 4806.509395] Lustre: server umount lustre-MDT0000 complete [ 4822.878604] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4822.879891] LDISKFS-fs (dm-0): recovery complete [ 4822.883287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4823.093977] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4823.097485] Lustre: Skipped 10 previous similar messages [ 4825.139453] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4828.295901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 4828.295901] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 4831.237634] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4832.108826] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4860.779429] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:18:45 (1781626725) [ 4865.605662] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4866.602835] Lustre: Failing over lustre-MDT0000 [ 4866.794526] Lustre: server umount lustre-MDT0000 complete [ 4882.740568] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4882.742592] LDISKFS-fs (dm-0): recovery complete [ 4882.747443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4885.312760] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4888.110405] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 4888.111483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 4891.064756] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4891.936388] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4896.524988] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 4897.251724] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 4900.227685] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:19:24 (1781626764) [ 4906.592098] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 4967.913345] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4967.915316] Lustre: Skipped 1 previous similar message [ 4967.916654] LustreError: 6515:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992fa66cb800 x1868166049977600/t296352743435(0) o36->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:538/0 lens 66040/440 e 0 to 0 dl 1781626888 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5027.140976] Lustre: 11288:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fa6f1fc50 x1868166049977600/t296352743435(0) o36->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:597/0 lens 66040/440 e 0 to 0 dl 1781626947 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5031.000513] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5031.747091] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:21:36 (1781626896) [ 5038.473153] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5039.883201] Lustre: Failing over lustre-MDT0000 [ 5040.029824] Lustre: server umount lustre-MDT0000 complete [ 5055.951870] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5055.955927] LDISKFS-fs (dm-0): recovery complete [ 5055.960472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.148765] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5062.128325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 5062.129064] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 5064.843174] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5065.669446] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5071.096715] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:22:15 (1781626935) [ 5081.301170] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5086.071150] Lustre: Failing over lustre-OST0000 [ 5086.114327] Lustre: server umount lustre-OST0000 complete [ 5087.200330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.205894] Lustre: Skipped 33 previous similar messages [ 5100.786944] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5100.788469] LDISKFS-fs (dm-2): recovery complete [ 5100.800736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5100.877042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5100.880047] Lustre: Skipped 8 previous similar messages [ 5102.884107] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5102.886657] Lustre: Skipped 35 previous similar messages [ 5103.349132] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5115.081366] Lustre: Failing over lustre-OST0000 [ 5115.138994] Lustre: server umount lustre-OST0000 complete [ 5128.720952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5131.564165] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5136.232504] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5136.990330] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5171.878817] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:23:56 (1781627036) [ 5173.043035] Lustre: Failing over lustre-MDT0000 [ 5173.260382] Lustre: server umount lustre-MDT0000 complete [ 5174.751882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5174.753434] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781627040 with bad export cookie 17193890692565240812 [ 5174.755794] LustreError: Skipped 6 previous similar messages [ 5174.762853] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5174.771698] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 130 previous similar messages [ 5187.038484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5188.858911] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5190.970289] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5190.974189] Lustre: Skipped 9 previous similar messages [ 5192.681427] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5192.691048] Lustre: Skipped 9 previous similar messages [ 5192.712930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 5192.713370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 5200.757470] Lustre: Failing over lustre-MDT0000 [ 5200.892149] Lustre: server umount lustre-MDT0000 complete [ 5214.644715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.705595] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5214.710988] LustreError: Skipped 6 previous similar messages [ 5216.733809] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5220.339830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 5220.340112] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 5222.355167] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5223.149429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5227.188706] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:24:51 (1781627091) [ 5238.893783] Lustre: Failing over lustre-OST0000 [ 5238.952800] Lustre: server umount lustre-OST0000 complete [ 5252.697510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5255.446882] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5260.216459] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5260.934739] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5265.684812] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:25:29 (1781627129) [ 5266.502304] Lustre: Failing over lustre-MDT0000 [ 5266.754128] Lustre: server umount lustre-MDT0000 complete [ 5270.783739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.838363] Lustre: *** cfs_fail_loc=605, val=0*** [ 5270.840387] LustreError: 164688:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0a0f480 failed: rc = -95 [ 5270.844890] LustreError: 164688:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 5270.846927] LustreError: 164688:0:(obd_mount.c:250:lustre_start_simple()) MGS setup error -95 [ 5270.849845] LustreError: 164688:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 5270.852202] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5270.854521] LustreError: 164688:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5270.861466] Lustre: server umount lustre-MDT0000 complete [ 5270.863257] LustreError: 164688:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5273.921708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5275.631150] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5279.224471] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 5279.224589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 5279.585290] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 12:25:43 (1781627143) [ 5282.685511] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5283.900233] Lustre: Failing over lustre-MDT0000 [ 5284.159319] Lustre: server umount lustre-MDT0000 complete [ 5299.114030] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5299.115700] LDISKFS-fs (dm-0): recovery complete [ 5299.118956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5300.972834] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5303.622994] Lustre: *** cfs_fail_loc=707, val=0*** [ 5364.033458] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 5364.053864] Lustre: 166857:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5364.059125] Lustre: 166857:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 1 previous similar message [ 5364.282912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 5364.282934] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 5367.235805] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5368.007454] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5372.383842] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 12:27:16 (1781627236) [ 5396.570203] LustreError: 6515:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff992fbfd85880 x1868166050859904/t0(0) o101->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:168/0 lens 664/0 e 0 to 0 dl 1781627273 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5396.577770] LustreError: 6515:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 5407.615100] LustreError: 6515:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5407.623418] LustreError: 39767:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff992fabeb7b80 x1868166050861184/t0(0) o35->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:179/0 lens 392/0 e 0 to 0 dl 1781627284 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5412.833136] LustreError: 8532:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff992fabf6d500 x1868166061997312/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:184/0 lens 544/0 e 0 to 0 dl 1781627289 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 5412.851142] LustreError: 8532:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 5420.144230] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 12:28:04 (1781627284) [ 5444.731244] LustreError: 42743:0:(tgt_handler.c:2836:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 5455.767164] LustreError: 42743:0:(tgt_handler.c:2836:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 5459.222713] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 12:28:43 (1781627323) [ 5482.336353] LustreError: 6518:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff992fabf06d80 x1868166050884224/t0(0) o101->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:253/0 lens 576/0 e 0 to 0 dl 1781627358 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5482.345829] LustreError: 6518:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 5487.447089] LustreError: 6518:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5487.455136] LustreError: 6515:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff992f850d4e00 x1868166050884864/t0(0) o101->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:258/0 lens 664/0 e 0 to 0 dl 1781627363 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5498.584286] LustreError: 6518:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5498.593224] LustreError: 8468:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff992f9f2ea680 x1868166050906752/t0(0) o101->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:309/0 lens 664/0 e 0 to 0 dl 1781627414 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5498.601823] LustreError: 8468:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 5515.604060] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 12:29:39 (1781627379) [ 5616.629763] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 12:31:20 (1781627480) [ 5642.060468] LustreError: 8468:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff992fc1f76300 x1868166050964352/t0(0) o101->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:413/0 lens 576/0 e 0 to 0 dl 1781627518 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5642.077510] LustreError: 8468:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 5642.081642] LustreError: 8468:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5642.086136] LustreError: 8468:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 5642.503120] LustreError: 8468:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5658.150778] LustreError: 6519:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5658.155160] LustreError: 6519:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 5658.575096] LustreError: 6519:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5658.580774] LustreError: 6519:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 5674.323600] LustreError: 8468:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff992fbfc22680 x1868166050983296/t0(0) o101->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:445/0 lens 576/0 e 0 to 0 dl 1781627550 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 5674.341027] LustreError: 8468:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 79 previous similar messages [ 5689.691954] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 12:32:33 (1781627553) [ 5719.552551] Lustre: DEBUG MARKER: phase 2 [ 5725.183353] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 12:33:09 (1781627589) [ 5801.965951] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 12:34:25 (1781627665) [ 5803.124796] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 5804.471821] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 12:34:28 (1781627668) [ 5808.378490] Lustre: DEBUG MARKER: Started rundbench load pid=135129 ... [ 5814.998316] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5817.074359] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 5818.309108] Lustre: Failing over lustre-MDT0000 [ 5818.608494] Lustre: server umount lustre-MDT0000 complete [ 5819.871577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5819.879227] LustreError: Skipped 2 previous similar messages [ 5819.888913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.903988] Lustre: Skipped 21 previous similar messages [ 5819.908233] LustreError: 151578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5819.922462] LustreError: 151578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 64 previous similar messages [ 5836.512068] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781627686/real 1781627686] req@ffff992fa66c8000 x1868166062268800/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781627702 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5836.529438] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.537238] LustreError: Skipped 2 previous similar messages [ 5837.268489] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5837.271360] LDISKFS-fs (dm-0): recovery complete [ 5837.282229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5847.008866] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fc1f74a80 x1868166062424320/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5847.146388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5847.152049] Lustre: Skipped 6 previous similar messages [ 5847.181179] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5847.186203] Lustre: Skipped 9 previous similar messages [ 5847.866779] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5847.870818] Lustre: Skipped 4 previous similar messages [ 5849.845340] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5852.645258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5852.657228] Lustre: Skipped 21 previous similar messages [ 5852.699905] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5852.705459] Lustre: Skipped 4 previous similar messages [ 5852.726115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 5852.726612] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 5856.875509] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5857.956521] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5865.847513] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5867.922682] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 5869.291721] Lustre: Failing over lustre-MDT0001 [ 5869.319284] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.8@tcp (stopping) [ 5869.469365] Lustre: server umount lustre-MDT0001 complete [ 5888.061121] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 5888.063594] LDISKFS-fs (dm-1): recovery complete [ 5888.075483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5891.190117] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5894.000692] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:571 to 0x2c0000400:609) [ 5894.004886] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:589 to 0x280000400:609) [ 5898.165030] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5899.366253] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5907.396123] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5909.623435] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 5911.066952] Lustre: Failing over lustre-MDT0000 [ 5911.251878] Lustre: server umount lustre-MDT0000 complete [ 5930.315632] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5930.319574] LDISKFS-fs (dm-0): recovery complete [ 5930.329051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5933.739125] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5936.165866] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 5936.166302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 5941.500307] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5942.823560] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5957.253931] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 12:37:00 (1781627820) [ 6083.448229] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6095.104249] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 6096.394195] Lustre: Failing over lustre-MDT0001 [ 6096.483771] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.8@tcp (stopping) [ 6099.941674] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6099.949449] Lustre: Skipped 1 previous similar message [ 6102.229590] Lustre: server umount lustre-MDT0001 complete [ 6119.290834] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6119.292949] LDISKFS-fs (dm-1): recovery complete [ 6119.302547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6121.521239] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6127.292580] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1111 to 0x280000400:1153) [ 6127.293120] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1112 to 0x2c0000400:1153) [ 6130.497765] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6131.399360] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6257.867190] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6269.299311] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 6270.433183] Lustre: Failing over lustre-MDT0000 [ 6270.510253] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 6270.514742] Lustre: Skipped 1 previous similar message [ 6276.264595] Lustre: server umount lustre-MDT0000 complete [ 6293.331292] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6293.333195] LDISKFS-fs (dm-0): recovery complete [ 6293.339932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6293.473834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 6293.478930] Lustre: Skipped 8 previous similar messages [ 6295.820287] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6300.690663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5891 to 0x280000401:5921) [ 6300.698155] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5860 to 0x2c0000401:5889) [ 6303.607161] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6304.669951] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6334.705631] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 12:43:18 (1781628198) [ 6459.750997] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6470.886778] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 6471.910688] Lustre: Failing over lustre-MDT0001 [ 6471.926130] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.8@tcp (stopping) [ 6472.151464] Lustre: server umount lustre-MDT0001 complete [ 6472.676463] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6472.680814] Lustre: Skipped 17 previous similar messages [ 6472.682461] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6472.686302] LustreError: Skipped 4 previous similar messages [ 6472.688049] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6472.702364] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 93 previous similar messages [ 6488.299399] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6488.303275] LDISKFS-fs (dm-1): recovery complete [ 6488.317064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6488.505996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6488.510976] Lustre: Skipped 4 previous similar messages [ 6488.525386] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6488.528226] Lustre: Skipped 4 previous similar messages [ 6489.402922] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6489.407261] Lustre: Skipped 4 previous similar messages [ 6490.644658] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6493.667512] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6493.671079] Lustre: Skipped 17 previous similar messages [ 6494.906185] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6494.914741] Lustre: Skipped 4 previous similar messages [ 6494.941120] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1692 to 0x280000400:1729) [ 6494.941120] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1692 to 0x2c0000400:1729) [ 6497.815179] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6498.783630] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6624.374117] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6635.505385] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 6636.435922] Lustre: Failing over lustre-MDT0000 [ 6636.445972] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 6642.283132] Lustre: server umount lustre-MDT0000 complete [ 6658.725136] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6658.726654] LDISKFS-fs (dm-0): recovery complete [ 6658.730822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.793335] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6658.798721] LustreError: Skipped 2 previous similar messages [ 6660.849814] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6665.554706] Lustre: 6517:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fc0eef800 x1868166073997952/t330712495362(0) o36->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:682/0 lens 552/2880 e 0 to 0 dl 1781628542 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6665.570844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6258 to 0x280000401:6273) [ 6665.571074] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6226 to 0x2c0000401:6241) [ 6668.183871] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6668.956101] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6673.538462] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 12:48:57 (1781628537) [ 6797.199368] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6808.080540] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 6808.835149] Lustre: Failing over lustre-MDT0001 [ 6808.849417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6808.851693] Lustre: Skipped 9 previous similar messages [ 6809.004189] Lustre: server umount lustre-MDT0001 complete [ 6823.640559] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6823.642425] LDISKFS-fs (dm-1): recovery complete [ 6823.651401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6825.167740] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6829.048327] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1901 to 0x2c0000400:1921) [ 6829.050199] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1901 to 0x280000400:1921) [ 6831.193887] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6832.001248] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6956.306671] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6967.081358] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 6967.766886] Lustre: Failing over lustre-MDT0001 [ 6967.777620] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6967.968136] Lustre: server umount lustre-MDT0001 complete [ 6982.555515] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6982.557114] LDISKFS-fs (dm-1): recovery complete [ 6982.562650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.173606] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6987.775691] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1901 to 0x2c0000400:1953) [ 6987.775692] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1901 to 0x280000400:1953) [ 6989.932694] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6990.608151] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6994.205566] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 12:54:18 (1781628858) [ 7000.237218] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7001.777727] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7002.432836] Lustre: Failing over lustre-OST0000 [ 7002.470667] Lustre: server umount lustre-OST0000 complete [ 7016.917260] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7016.918658] LDISKFS-fs (dm-2): recovery complete [ 7016.921666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7019.049128] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7023.078759] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7023.731178] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7032.746470] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7034.310502] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 7035.011089] Lustre: Failing over lustre-OST0000 [ 7035.055284] Lustre: server umount lustre-OST0000 complete [ 7049.377115] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7049.378902] LDISKFS-fs (dm-2): recovery complete [ 7049.382466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7051.519112] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7055.480769] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7056.103594] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7061.670081] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 12:55:26 (1781628926) [ 7184.720372] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7187.445805] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7198.239845] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 7198.915232] Lustre: Failing over lustre-MDT0001 [ 7198.920781] LustreError: lustre-MDT0001-osp-MDT0000: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 7198.924086] LustreError: Skipped 4 previous similar messages [ 7198.925584] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7198.930044] Lustre: Skipped 16 previous similar messages [ 7198.932365] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7198.934984] Lustre: Skipped 3 previous similar messages [ 7199.111850] Lustre: server umount lustre-MDT0001 complete [ 7200.453948] Lustre: Failing over lustre-MDT0000 [ 7200.674614] Lustre: server umount lustre-MDT0000 complete [ 7215.645784] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7215.649326] LDISKFS-fs (dm-1): recovery complete [ 7215.657456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7215.741229] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7215.743372] LDISKFS-fs (dm-0): recovery complete [ 7215.750648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7215.808782] LustreError: 196755:0:(llog.c:1646:llog_backup()) MGC192.168.202.108@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7215.808920] LustreError: 196735:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGC192.168.202.108@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff992fa0244400) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7215.812044] Lustre: 196755:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.108@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7215.877091] LustreError: 8452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7215.884687] LustreError: 8452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 69 previous similar messages [ 7215.909671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7215.911864] Lustre: Skipped 5 previous similar messages [ 7215.923642] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7215.929277] Lustre: Skipped 5 previous similar messages [ 7216.011108] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7216.011996] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7216.014321] Lustre: Skipped 5 previous similar messages [ 7216.017493] Lustre: Skipped 16 previous similar messages [ 7217.564749] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7217.643476] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7219.103143] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781629068/real 1781629068] req@ffff992e88537b80 x1868166084283776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781629084 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7219.114057] Lustre: 3647:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7221.389262] LustreError: 8452:0:(ldlm_lockd.c:915:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-lustre-MDT0001_UUID lock: ffff992e8873a400/0xee9cf87cdd7a5711 lrc: 3/0,0 mode: PW/PW res: [0x240001f59:0xdbf:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x40000000000020 nid: 0@lo remote: 0xee9cf87cdd7a570a expref: 51 pid: 182632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 7222.054248] Lustre: lustre-MDT0001: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 7222.057627] Lustre: Skipped 5 previous similar messages [ 7222.073617] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1901 to 0x2c0000400:1985) [ 7222.073637] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1901 to 0x280000400:1985) [ 7226.412503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:8769) [ 7226.412533] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8705 to 0x2c0000401:8737) [ 7228.261617] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7228.830118] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7229.366195] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7353.744204] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7356.710539] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7367.524797] Lustre: DEBUG MARKER: fail mds2 mds1 2 times [ 7368.231229] Lustre: Failing over lustre-MDT0001 [ 7368.416528] LustreError: 196761:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000085c2aa8f ns: mdt-lustre-MDT0001_UUID lock: ffff992e85bf8800/0xee9cf87cdd866b20 lrc: 3/0,0 mode: PR/PR res: [0x2400036c8:0xf2a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.202.8@tcp remote: 0x11f02f6e637564f4 expref: 3 pid: 196761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 7368.575201] Lustre: server umount lustre-MDT0001 complete [ 7369.247125] Lustre: 3649:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781629073/real 1781629073] req@ffff992e88af6680 x1868166084284416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781629089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7369.258041] Lustre: 3649:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 7369.956214] Lustre: Failing over lustre-MDT0000 [ 7370.147833] Lustre: server umount lustre-MDT0000 complete [ 7385.323374] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7385.327827] LDISKFS-fs (dm-1): recovery complete [ 7385.333194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.390377] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7385.392171] LDISKFS-fs (dm-0): recovery complete [ 7385.395325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.457376] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7385.462480] LustreError: Skipped 1 previous similar message [ 7385.464289] LustreError: 200121:0:(llog.c:1646:llog_backup()) MGC192.168.202.108@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7385.464428] LustreError: 200118:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGC192.168.202.108@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff992f89776e00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7385.467525] Lustre: 200121:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.108@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7387.215545] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7387.338912] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7391.026243] LustreError: 8452:0:(ldlm_lockd.c:915:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-lustre-MDT0001_UUID lock: ffff992e926aa000/0xee9cf87cdd866e8b lrc: 3/0,0 mode: PW/PW res: [0x2400036c8:0xdc5:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x40000000000020 nid: 0@lo remote: 0xee9cf87cdd866e84 expref: 51 pid: 182632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 7391.711158] Lustre: 3650:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781629241/real 1781629241] req@ffff992fa66e1f80 x1868166093444864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781629257 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7391.725155] Lustre: 3650:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 7391.746339] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1901 to 0x280000400:2017) [ 7391.746412] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1901 to 0x2c0000400:2017) [ 7397.167872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:8801) [ 7397.168100] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8705 to 0x2c0000401:8769) [ 7399.358493] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7400.070208] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7400.761711] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7404.466620] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 13:01:08 (1781629268) [ 7407.299667] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7408.273135] Lustre: Failing over lustre-MDT0000 [ 7408.453131] Lustre: server umount lustre-MDT0000 complete [ 7422.821431] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7422.822710] LDISKFS-fs (dm-0): recovery complete [ 7422.825214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7424.315927] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7424.364327] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7440.712762] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 7440.729433] Lustre: 200127:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992e89230a80 x1868166085298688/t335007474511(335007474511) o101->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:702/0 lens 520/3488 e 0 to 0 dl 1781629317 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7440.760220] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8771 to 0x2c0000401:8801) [ 7440.760240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:8833) [ 7442.755356] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7443.319148] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7446.741488] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 13:01:51 (1781629311) [ 7449.258218] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7450.195085] Lustre: Failing over lustre-MDT0000 [ 7450.388647] Lustre: server umount lustre-MDT0000 complete [ 7464.584082] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7464.585284] LDISKFS-fs (dm-0): recovery complete [ 7464.587675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.288508] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7465.290740] LustreError: 200128:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992e896fa300 x1868166085298688/t335007474511(335007474511) o101->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:726/0 lens 520/664 e 0 to 0 dl 1781629341 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7466.317714] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7481.665525] Lustre: lustre-MDT0000: Client 13fc764b-ac08-40d4-bea6-629cf6c12fe2 (at 192.168.202.8@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 7481.670223] Lustre: 202055:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992e926ad180 x1868166085298688/t335007474511(335007474511) o101->13fc764b-ac08-40d4-bea6-629cf6c12fe2@192.168.202.8@tcp:743/0 lens 520/3488 e 0 to 0 dl 1781629358 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7481.703712] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8803 to 0x2c0000401:8833) [ 7481.704176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:8865) [ 7483.672328] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7484.256417] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7487.626670] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 13:02:32 (1781629352) [ 7489.020990] Lustre: Failing over lustre-OST0000 [ 7489.068306] Lustre: server umount lustre-OST0000 complete [ 7490.327102] Lustre: Failing over lustre-MDT0000 [ 7490.420633] Lustre: server umount lustre-MDT0000 complete [ 7503.362632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7504.901787] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7505.626208] Lustre: lustre-MDT0000: Denying connection for new client 21298288-242c-4b68-b297-83fc79265999 (at 192.168.202.8@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7508.986187] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8803 to 0x2c0000401:8865) [ 7512.974703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7515.138221] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7518.181555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:8897) [ 7519.170355] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 13:03:03 (1781629383) [ 7519.513703] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7519.515608] LustreError: 182632:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992fc01dd500 x1868166094587776/t356482285575(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:26/0 lens 1312/4320 e 0 to 0 dl 1781629396 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7521.927326] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7522.537123] Lustre: Failing over lustre-MDT0000 [ 7522.769578] Lustre: server umount lustre-MDT0000 complete [ 7536.095145] Lustre: 200155:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781629385/real 1781629385] req@ffff992fbcc48700 x1868166094587776/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1781629401 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7536.103221] Lustre: 200155:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 7536.881426] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7536.883026] LDISKFS-fs (dm-0): recovery complete [ 7536.885918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.418846] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7542.276539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:8929) [ 7542.276638] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:8897) [ 7544.121296] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7544.687893] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7548.381222] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 13:03:32 (1781629412) [ 7551.391704] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7554.135908] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7554.826195] Lustre: Failing over lustre-MDT0001 [ 7554.832596] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.8@tcp (stopping) [ 7554.836116] Lustre: Skipped 2 previous similar messages [ 7561.281321] Lustre: server umount lustre-MDT0001 complete [ 7575.363881] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7575.365136] LDISKFS-fs (dm-1): recovery complete [ 7575.368824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.741304] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7580.658856] Lustre: 202055:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fbc804e00 x1868166097433728/t34359740369(0) o36->21298288-242c-4b68-b297-83fc79265999@192.168.202.8@tcp:108/0 lens 560/2880 e 0 to 0 dl 1781629478 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7580.662205] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2028 to 0x280000400:2049) [ 7580.663181] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2028 to 0x2c0000400:2049) [ 7582.356693] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7582.888611] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7586.411387] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 13:04:10 (1781629450) [ 7586.756588] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7586.757975] Lustre: Skipped 1 previous similar message [ 7586.759185] LustreError: 182632:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992e8777fb80 x1868166094648960/t360777252897(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:93/0 lens 2520/4320 e 0 to 0 dl 1781629463 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7586.765704] LustreError: 182632:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 1 previous similar message [ 7589.416244] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7592.076602] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7592.782782] Lustre: Failing over lustre-MDT0000 [ 7593.016085] Lustre: server umount lustre-MDT0000 complete [ 7607.664578] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7607.666597] LDISKFS-fs (dm-0): recovery complete [ 7607.672918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.507870] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7612.952576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:8961) [ 7612.953311] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:8929) [ 7614.918648] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7615.537564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7617.497781] Lustre: Failing over lustre-MDT0001 [ 7617.581534] Lustre: server umount lustre-MDT0001 complete [ 7631.987733] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7631.989387] LDISKFS-fs (dm-1): recovery complete [ 7631.993819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.657175] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7637.495200] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2060 to 0x280000400:2081) [ 7637.495405] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2060 to 0x2c0000400:2081) [ 7639.630789] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7640.314285] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7644.180621] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:05:08 (1781629508) [ 7650.396312] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7653.333084] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7654.133231] Lustre: Failing over lustre-MDT0000 [ 7654.378924] Lustre: server umount lustre-MDT0000 complete [ 7655.789536] LustreError: 151745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629521 with bad export cookie 17193890692569979749 [ 7655.790511] Lustre: Failing over lustre-MDT0001 [ 7655.793898] LustreError: 151745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7662.048167] LustreError: 216639:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff992fa660b480 x1868166094712832/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7662.054688] LustreError: 216639:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7662.174981] Lustre: server umount lustre-MDT0001 complete [ 7677.415292] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7677.417599] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7677.418449] LDISKFS-fs (dm-0): recovery complete [ 7677.419975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7677.420489] LDISKFS-fs (dm-1): recovery complete [ 7677.429719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7680.479397] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fb6e64380 x1868166094713600/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7682.311517] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7682.415056] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7685.960934] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2092 to 0x2c0000400:2113) [ 7685.961225] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2092 to 0x280000400:2113) [ 7685.965861] Lustre: 217755:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992e8a3e5500 x1868166097500800/t42949673028(0) o36->21298288-242c-4b68-b297-83fc79265999@192.168.202.8@tcp:217/0 lens 560/2880 e 0 to 0 dl 1781629587 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7698.269829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:8993) [ 7698.269843] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:8961) [ 7700.391642] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7700.969132] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7701.585619] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7705.395361] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:06:09 (1781629569) [ 7705.773512] LustreError: 217754:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992e8d6c0700 x1868166097534208/t47244640326(0) o36->21298288-242c-4b68-b297-83fc79265999@192.168.202.8@tcp:251/0 lens 560/448 e 0 to 0 dl 1781629621 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7705.784339] LustreError: 217754:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 1 previous similar message [ 7713.227036] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7714.569905] Lustre: Failing over lustre-MDT0000 [ 7714.828409] Lustre: server umount lustre-MDT0000 complete [ 7733.217056] Lustre: 3648:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781629582/real 1781629582] req@ffff992fc0a98a80 x1868166094758272/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781629598 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7733.243908] Lustre: 3648:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7734.406868] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7734.409275] LDISKFS-fs (dm-0): recovery complete [ 7734.414838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7746.966676] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7748.646346] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:8993) [ 7748.652218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:9025) [ 7754.493716] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7755.757136] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7760.280142] Lustre: lustre-MDT0001: Client 21298288-242c-4b68-b297-83fc79265999 (at 192.168.202.8@tcp) reconnecting [ 7760.286857] Lustre: Skipped 2 previous similar messages [ 7760.298349] Lustre: 218524:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fbff59c00 x1868166097534208/t47244640326(0) o36->21298288-242c-4b68-b297-83fc79265999@192.168.202.8@tcp:305/0 lens 560/2880 e 0 to 0 dl 1781629675 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7765.512973] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:07:09 (1781629629) [ 7766.182929] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7766.185959] Lustre: Skipped 2 previous similar messages [ 7770.533973] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7771.584554] Lustre: Failing over lustre-MDT0001 [ 7771.741979] Lustre: server umount lustre-MDT0001 complete [ 7787.932348] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7787.937979] LDISKFS-fs (dm-1): recovery complete [ 7787.958582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7790.312802] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7793.664649] Lustre: 217755:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992e898f9880 x1868166097568256/t47244640397(0) o36->21298288-242c-4b68-b297-83fc79265999@192.168.202.8@tcp:300/0 lens 560/2880 e 0 to 0 dl 1781629670 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7793.674133] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2134 to 0x2c0000400:2177) [ 7793.678958] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2134 to 0x280000400:2177) [ 7796.428765] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7797.334410] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7802.365878] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:07:46 (1781629666) [ 7810.160770] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7814.320183] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7815.387678] Lustre: Failing over lustre-MDT0000 [ 7815.637569] Lustre: server umount lustre-MDT0000 complete [ 7816.524388] LustreError: 218524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7816.533491] LustreError: 218524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 156 previous similar messages [ 7819.074444] Lustre: 218524:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fc062fb80 x1868166097598720/t51539607622(0) o36->21298288-242c-4b68-b297-83fc79265999@192.168.202.8@tcp:325/0 lens 560/2880 e 0 to 0 dl 1781629695 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7819.231657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7819.232462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.236341] LustreError: Skipped 10 previous similar messages [ 7819.239589] Lustre: Skipped 47 previous similar messages [ 7832.213092] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7832.219358] LDISKFS-fs (dm-0): recovery complete [ 7832.223213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7832.389409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7832.394434] Lustre: Skipped 15 previous similar messages [ 7832.415478] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7832.419434] Lustre: Skipped 15 previous similar messages [ 7832.893067] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7832.898286] Lustre: Skipped 15 previous similar messages [ 7834.367332] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7837.668809] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7837.673438] Lustre: Skipped 51 previous similar messages [ 7837.709781] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7837.713496] Lustre: Skipped 15 previous similar messages [ 7837.742247] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9025) [ 7837.742988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:9057) [ 7840.461948] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7841.370740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7843.772140] Lustre: Failing over lustre-MDT0001 [ 7843.894375] Lustre: server umount lustre-MDT0001 complete [ 7859.982656] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7859.984475] LDISKFS-fs (dm-1): recovery complete [ 7859.993472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7862.299771] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7865.351569] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2188 to 0x2c0000400:2209) [ 7865.356252] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2188 to 0x280000400:2209) [ 7868.063782] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7869.023600] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7874.301627] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:08:58 (1781629738) [ 7874.862078] LustreError: 217754:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992e876c1500 x1868166097637760/t55834574918(0) o36->21298288-242c-4b68-b297-83fc79265999@192.168.202.8@tcp:381/0 lens 560/448 e 0 to 0 dl 1781629751 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7874.878480] LustreError: 217754:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 2 previous similar messages [ 7882.453139] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7887.095649] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7888.217731] Lustre: Failing over lustre-MDT0000 [ 7888.444476] Lustre: server umount lustre-MDT0000 complete [ 7890.239320] Lustre: lustre-MDT0001: Client 21298288-242c-4b68-b297-83fc79265999 (at 192.168.202.8@tcp) reconnecting [ 7890.242276] Lustre: Skipped 1 previous similar message [ 7890.246074] Lustre: 217755:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fbcb8ad80 x1868166097637760/t55834574918(0) o36->21298288-242c-4b68-b297-83fc79265999@192.168.202.8@tcp:396/0 lens 560/2880 e 0 to 0 dl 1781629766 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7890.745276] LustreError: 151745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629756 with bad export cookie 17193890692569993518 [ 7890.748860] Lustre: Failing over lustre-MDT0001 [ 7890.751675] LustreError: 151745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7890.938048] Lustre: server umount lustre-MDT0001 complete [ 7907.295566] Lustre: 3648:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781629756/real 1781629756] req@ffff992e898f8000 x1868166094891392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781629772 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7907.318828] Lustre: 3648:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7908.825994] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7908.829693] LDISKFS-fs (dm-0): recovery complete [ 7908.840996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.867030] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7908.871625] LDISKFS-fs (dm-1): recovery complete [ 7908.881897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7915.551822] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fc0138700 x1868166094893952/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7918.679254] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7918.833436] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7922.010975] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2220 to 0x280000400:2241) [ 7922.011066] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2220 to 0x2c0000400:2241) [ 7930.257128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:9089) [ 7930.262646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9057) [ 7933.698366] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7934.745639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7935.676910] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7941.209629] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:10:05 (1781629805) [ 7946.092888] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7947.174390] Lustre: Failing over lustre-MDT0001 [ 7947.178331] LustreError: 230748:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x240003e99:0xa9:0x0].0x0 (ffff992f92d0a500) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7952.955870] Lustre: server umount lustre-MDT0001 complete [ 7968.177848] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7968.180040] LDISKFS-fs (dm-1): recovery complete [ 7968.185498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7969.873326] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7973.356439] Lustre: 228933:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fc0d94000 x1868166097675904/t60129542216(0) o36->21298288-242c-4b68-b297-83fc79265999@192.168.202.8@tcp:479/0 lens 496/2888 e 0 to 0 dl 1781629849 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 7973.362630] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2252 to 0x2c0000400:2273) [ 7973.362638] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2252 to 0x280000400:2273) [ 7975.274288] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7975.948648] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7979.665849] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:10:44 (1781629844) [ 7982.855403] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7983.491974] Lustre: Failing over lustre-MDT0000 [ 7983.687352] Lustre: server umount lustre-MDT0000 complete [ 7997.647368] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7997.649472] LDISKFS-fs (dm-0): recovery complete [ 7997.651618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7997.685347] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7997.690275] LustreError: Skipped 9 previous similar messages [ 7999.083994] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8003.066778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:9121) [ 8003.066790] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9089) [ 8004.815189] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8005.341184] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8008.762944] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:11:13 (1781629873) [ 8011.563556] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8014.016096] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8014.661492] Lustre: Failing over lustre-MDT0000 [ 8014.999882] Lustre: server umount lustre-MDT0000 complete [ 8029.466316] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8029.468483] LDISKFS-fs (dm-0): recovery complete [ 8029.472335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8031.058511] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8034.814688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:9153) [ 8034.814777] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9121) [ 8036.622553] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8037.195297] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8039.241566] Lustre: Failing over lustre-MDT0001 [ 8039.338878] Lustre: server umount lustre-MDT0001 complete [ 8053.372074] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8053.373334] LDISKFS-fs (dm-1): recovery complete [ 8053.376664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8054.771781] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8058.871187] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2252 to 0x280000400:2305) [ 8058.871186] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2252 to 0x2c0000400:2305) [ 8060.671881] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8061.250594] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8064.644543] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:12:09 (1781629929) [ 8065.017173] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8065.018335] Lustre: Skipped 5 previous similar messages [ 8067.388931] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8069.758031] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8070.359873] Lustre: Failing over lustre-MDT0000 [ 8070.458533] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 8070.461293] Lustre: Skipped 22 previous similar messages [ 8070.608489] Lustre: server umount lustre-MDT0000 complete [ 8071.797863] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629937 with bad export cookie 17193890692570003976 [ 8071.799156] Lustre: Failing over lustre-MDT0001 [ 8071.803170] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8077.791792] LustreError: 238990:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff992fb6e9fb80 x1868166095039104/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8077.797051] LustreError: 238990:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8077.913084] Lustre: server umount lustre-MDT0001 complete [ 8092.429430] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8092.431437] LDISKFS-fs (dm-1): recovery complete [ 8092.434083] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8092.437273] LDISKFS-fs (dm-0): recovery complete [ 8092.437381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8092.439828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8092.525474] LustreError: 240087:0:(llog.c:1646:llog_backup()) MGC192.168.202.108@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8092.527663] Lustre: 240087:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.108@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8096.671362] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fb6e9dc00 x1868166095040256/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8098.211524] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8098.311550] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8100.994905] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9153) [ 8100.994935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:9185) [ 8101.013400] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2252 to 0x280000400:2337) [ 8101.013511] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2252 to 0x2c0000400:2337) [ 8102.790258] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8103.311459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8103.794502] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8107.096949] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:12:51 (1781629971) [ 8110.165373] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8110.780104] Lustre: Failing over lustre-MDT0000 [ 8110.942885] Lustre: server umount lustre-MDT0000 complete [ 8124.920451] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8124.921701] LDISKFS-fs (dm-0): recovery complete [ 8124.925482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8126.327837] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8130.559408] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9185) [ 8130.559433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:9217) [ 8132.377966] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8132.937240] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8162.617878] Lustre: lustre-MDT0001: Client 21298288-242c-4b68-b297-83fc79265999 (at 192.168.202.8@tcp) reconnecting [ 8162.623557] Lustre: 240099:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff992fbcb7c380 x1868166097731968/t73014444034(0) o36->21298288-242c-4b68-b297-83fc79265999@192.168.202.8@tcp:708/0 lens 496/2888 e 0 to 0 dl 1781630078 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8162.632983] Lustre: 240099:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8164.876620] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 13:13:49 (1781630029) [ 8165.213317] LustreError: 241961:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff992fa66c9500 x1868166097747456/t73014444037(0) o36->21298288-242c-4b68-b297-83fc79265999@192.168.202.8@tcp:671/0 lens 496/456 e 0 to 0 dl 1781630041 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8165.224982] LustreError: 241961:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 5 previous similar messages [ 8167.644471] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8168.310279] Lustre: Failing over lustre-MDT0001 [ 8168.421291] Lustre: server umount lustre-MDT0001 complete [ 8182.361483] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8182.362877] LDISKFS-fs (dm-1): recovery complete [ 8182.366423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8183.761143] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8187.893288] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2252 to 0x280000400:2369) [ 8187.893288] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2252 to 0x2c0000400:2369) [ 8189.617416] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8190.116195] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8193.485937] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 13:14:17 (1781630057) [ 8196.326284] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8198.804482] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8199.423854] Lustre: Failing over lustre-MDT0000 [ 8199.612739] Lustre: server umount lustre-MDT0000 complete [ 8213.543936] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8213.545836] LDISKFS-fs (dm-0): recovery complete [ 8213.549437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.026244] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8219.137981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9217) [ 8219.138078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:9249) [ 8220.971603] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8221.497449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8223.447665] Lustre: Failing over lustre-MDT0001 [ 8223.532261] Lustre: server umount lustre-MDT0001 complete [ 8237.696593] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8237.698095] LDISKFS-fs (dm-1): recovery complete [ 8237.702153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8239.190849] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8243.188744] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2252 to 0x280000400:2401) [ 8243.188748] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2252 to 0x2c0000400:2401) [ 8244.946320] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8245.554900] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8249.089211] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 13:15:13 (1781630113) [ 8252.013108] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8254.474853] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8255.088536] Lustre: Failing over lustre-MDT0000 [ 8255.276495] Lustre: server umount lustre-MDT0000 complete [ 8256.469732] LustreError: 151745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781630121 with bad export cookie 17193890692570011543 [ 8256.471309] Lustre: Failing over lustre-MDT0001 [ 8256.472795] LustreError: 151745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8256.577291] Lustre: server umount lustre-MDT0001 complete [ 8271.176120] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8271.177879] LDISKFS-fs (dm-0): recovery complete [ 8271.180483] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8271.181060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8271.182278] LDISKFS-fs (dm-1): recovery complete [ 8271.189967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8274.911215] Lustre: 3650:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781630124/real 1781630124] req@ffff992fc0591c00 x1868166095182720/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781630140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8274.922025] Lustre: 3650:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [ 8281.055592] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fc0591f80 x1868166095185408/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8282.624931] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8282.757374] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8285.527588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8736 to 0x280000401:9281) [ 8285.527587] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9249) [ 8290.635228] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2252 to 0x2c0000400:2433) [ 8290.635242] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2252 to 0x280000400:2433) [ 8293.002858] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8293.727021] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8294.428406] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8298.798556] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 13:16:03 (1781630163) [ 8299.653320] Lustre: 252680:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 21298288-242c-4b68-b297-83fc79265999 at adminstrative request [ 8304.533770] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 13:16:08 (1781630168) [ 8308.739280] Lustre: Failing over lustre-MDT0000 [ 8308.950517] Lustre: server umount lustre-MDT0000 complete [ 8322.271576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8323.678971] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8327.698907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9332 to 0x280000401:9377) [ 8327.698920] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9301 to 0x2c0000401:9345) [ 8329.447462] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8329.989414] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8333.187059] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 13:16:37 (1781630197) [ 8336.966440] Lustre: Failing over lustre-OST0000 [ 8337.030461] Lustre: server umount lustre-OST0000 complete [ 8349.867484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8351.891742] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8355.601047] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8356.165723] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8359.765319] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 13:17:04 (1781630224) [ 8361.083097] Lustre: Failing over lustre-MDT0000 [ 8361.247135] Lustre: server umount lustre-MDT0000 complete [ 8362.977187] LustreError: 151745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781630228 with bad export cookie 17193890692570024458 [ 8363.698830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.143981] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8365.882663] Lustre: lustre-MDT0000: Denying connection for new client 733991c5-29ea-4bee-837e-777e14c0dda5 (at 192.168.202.8@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 8365.886806] Lustre: Skipped 2 previous similar messages [ 8369.145373] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9301 to 0x2c0000401:9377) [ 8369.145465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9332 to 0x280000401:9409) [ 8373.185689] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 13:17:17 (1781630237) [ 8375.922416] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8376.750982] Lustre: Failing over lustre-OST0000 [ 8376.789949] Lustre: server umount lustre-OST0000 complete [ 8390.624965] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8390.626012] LDISKFS-fs (dm-2): recovery complete [ 8390.628374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8392.507116] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8396.036185] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8396.572453] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8400.062469] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 13:17:44 (1781630264) [ 8402.955531] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8404.827805] Lustre: Failing over lustre-OST0000 [ 8404.860360] Lustre: server umount lustre-OST0000 complete [ 8417.081409] LustreError: 178953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8417.085709] LustreError: 178953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 184 previous similar messages [ 8418.937325] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8418.938648] LDISKFS-fs (dm-2): recovery complete [ 8418.941968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8420.824544] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8421.004695] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.8@tcp inode [0x20002f9d1:0x5:0x0] object 0x280000401:9411 extent [0-4194303]: client csum 23229617, server csum d83645d6 [ 8424.427101] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8424.957276] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8428.184894] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 13:18:12 (1781630292) [ 8430.529464] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8432.848955] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8434.676905] Lustre: Failing over lustre-MDT0000 [ 8434.912728] Lustre: server umount lustre-MDT0000 complete [ 8435.679407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8435.683552] LustreError: Skipped 15 previous similar messages [ 8435.686022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8435.691781] Lustre: Skipped 59 previous similar messages [ 8436.052307] LustreError: 43913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781630301 with bad export cookie 17193890692570038689 [ 8436.054404] Lustre: Failing over lustre-OST0000 [ 8436.055126] LustreError: 43913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8436.090596] Lustre: server umount lustre-OST0000 complete [ 8450.544489] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8450.546382] LDISKFS-fs (dm-0): recovery complete [ 8450.549789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8461.279804] LustreError: 3646:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff992fb6e5a300 x1868166095333632/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8461.351171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8461.353996] Lustre: Skipped 20 previous similar messages [ 8461.377155] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8461.379731] Lustre: Skipped 20 previous similar messages [ 8462.675590] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8463.161286] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8463.165349] Lustre: Skipped 20 previous similar messages [ 8466.403852] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 8466.406048] Lustre: Skipped 66 previous similar messages [ 8466.496883] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8466.499296] Lustre: Skipped 20 previous similar messages [ 8466.514600] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9301 to 0x2c0000401:9409) [ 8476.900097] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8476.901813] LDISKFS-fs (dm-2): recovery complete [ 8476.905582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8478.755969] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8485.858922] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 13:19:10 (1781630350) [ 8490.569904] Lustre: Failing over lustre-OST0000 [ 8490.633149] Lustre: server umount lustre-OST0000 complete [ 8491.862408] Lustre: Failing over lustre-MDT0000 [ 8492.152918] Lustre: server umount lustre-MDT0000 complete [ 8504.955680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8506.459698] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8510.457248] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9301 to 0x2c0000401:9441) [ 8515.988069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8517.997869] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8518.715314] Lustre: lustre-OST0000: Denying connection for new client 84022035-581e-45a9-9b61-9e103e712a85 (at 192.168.202.8@tcp), waiting for 3 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 8524.089467] Lustre: lustre-OST0000: Denying connection for new client 84022035-581e-45a9-9b61-9e103e712a85 (at 192.168.202.8@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 8534.329554] Lustre: lustre-OST0000: Denying connection for new client 84022035-581e-45a9-9b61-9e103e712a85 (at 192.168.202.8@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 8534.333755] Lustre: Skipped 1 previous similar message [ 8554.811267] Lustre: lustre-OST0000: Denying connection for new client 84022035-581e-45a9-9b61-9e103e712a85 (at 192.168.202.8@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 8554.818163] Lustre: Skipped 3 previous similar messages [ 8587.500167] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 8587.502345] Lustre: 266218:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 733991c5-29ea-4bee-837e-777e14c0dda5@ [ 8587.506386] Lustre: 266218:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 8587.509149] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 8587.520061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9460 to 0x280000401:9481) [ 8591.869393] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 8600.259537] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 8602.760463] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 13:21:07 (1781630467) [ 8604.867246] Lustre: Failing over lustre-OST0000 [ 8604.932772] Lustre: server umount lustre-OST0000 complete [ 8620.399499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8624.099139] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8629.162906] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 13:21:33 (1781630493) [ 8631.050495] Lustre: Failing over lustre-OST0000 [ 8631.101929] Lustre: server umount lustre-OST0000 complete [ 8644.565968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8646.188115] LustreError: 269415:0:(ldlm_lib.c:2891:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 8646.203152] LustreError: 269415:0:(ldlm_lib.c:2891:target_recovery_thread()) Skipped 77 previous similar messages [ 8647.004690] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8652.255295] Lustre: *** cfs_fail_loc=715, val=40*** [ 8662.336361] Lustre: lustre-OST0000: Client 84022035-581e-45a9-9b61-9e103e712a85 (at 192.168.202.8@tcp) reconnected, waiting for 3 clients in recovery for 1:24 [ 8668.639181] Lustre: *** cfs_fail_loc=715, val=40*** [ 8668.640922] Lustre: Skipped 2 previous similar messages [ 8678.713287] Lustre: lustre-OST0000: Client 84022035-581e-45a9-9b61-9e103e712a85 (at 192.168.202.8@tcp) reconnected, waiting for 3 clients in recovery for 1:07 [ 8678.718116] Lustre: Skipped 2 previous similar messages [ 8685.023131] Lustre: *** cfs_fail_loc=715, val=40*** [ 8685.024702] Lustre: Skipped 2 previous similar messages [ 8686.247102] LustreError: 269415:0:(ldlm_lib.c:2891:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8686.250669] LustreError: 269415:0:(ldlm_lib.c:2891:target_recovery_thread()) Skipped 77 previous similar messages [ 8688.197139] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8688.776972] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8692.606183] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 13:22:37 (1781630557) [ 8694.167140] Lustre: Failing over lustre-MDT0000 [ 8694.443887] Lustre: server umount lustre-MDT0000 complete [ 8707.312165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8707.351238] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8707.354720] LustreError: Skipped 9 previous similar messages [ 8708.758954] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8712.677852] LustreError: 270968:0:(ldlm_lib.c:2891:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 8715.743145] Lustre: *** cfs_fail_loc=715, val=80*** [ 8715.744652] Lustre: Skipped 2 previous similar messages [ 8725.830640] Lustre: lustre-MDT0000: Client 84022035-581e-45a9-9b61-9e103e712a85 (at 192.168.202.8@tcp) reconnected, waiting for 2 clients in recovery for 1:27 [ 8725.835423] Lustre: Skipped 2 previous similar messages [ 8729.055527] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8732.127159] Lustre: *** cfs_fail_loc=715, val=80*** [ 8732.129123] Lustre: Skipped 1 previous similar message [ 8741.177457] Lustre: lustre-MDT0000: Client 84022035-581e-45a9-9b61-9e103e712a85 (at 192.168.202.8@tcp) reconnected, waiting for 2 clients in recovery for 1:12 [ 8745.439539] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8747.487150] Lustre: *** cfs_fail_loc=715, val=80*** [ 8747.489050] Lustre: Skipped 1 previous similar message [ 8757.561306] Lustre: lustre-MDT0000: Client 84022035-581e-45a9-9b61-9e103e712a85 (at 192.168.202.8@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 8760.799815] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8763.871141] Lustre: *** cfs_fail_loc=715, val=80*** [ 8763.872774] Lustre: Skipped 1 previous similar message [ 8773.945259] Lustre: lustre-MDT0000: Client 84022035-581e-45a9-9b61-9e103e712a85 (at 192.168.202.8@tcp) reconnected, waiting for 2 clients in recovery for 0:39 [ 8777.183496] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8792.543502] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8792.759088] LustreError: 270968:0:(ldlm_lib.c:2891:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8792.764240] Lustre: 270968:0:(ldlm_lib.c:2937:target_recovery_thread()) too long recovery - read logs [ 8792.768196] LustreError: dumping log to /tmp/lustre-log.1781630658.270968 [ 8792.828138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9495 to 0x280000401:9513) [ 8792.828139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9454 to 0x2c0000401:9473) [ 8794.701170] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8795.268600] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8798.775738] Lustre: DEBUG MARKER: == replay-single test complete, duration 8539 sec ======== 13:24:23 (1781630663) [ 8799.293747] Lustre: DEBUG MARKER: === replay-single: start cleanup 13:24:23 (1781630663) === [ 8801.768086] Lustre: DEBUG MARKER: === replay-single: finish cleanup 13:24:26 (1781630666) === [ 8802.420672] Lustre: Failing over lustre-MDT0000 [ 8802.537407] Lustre: server umount lustre-MDT0000 complete [ 8818.053733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8818.143844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 8818.146224] Lustre: Skipped 11 previous similar messages [ 8819.775145] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8823.290329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9495 to 0x280000401:9545) [ 8823.290355] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9454 to 0x2c0000401:9505) [ 8825.204485] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8825.801797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8838.420389] Lustre: server umount lustre-MDT0000 complete [ 8841.339560] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781630706 with bad export cookie 17193890692570071057 [ 8841.343508] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8841.438979] Lustre: server umount lustre-MDT0001 complete [ 8853.639864] Lustre: server umount lustre-OST0000 complete [ 8867.202483] Lustre: server umount lustre-OST0001 complete [ 8874.895669] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 8876.230858] Key type lgssc unregistered [ 8876.378478] LNet: 275783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8876.382366] LNetError: 275783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8876.394213] LNet: Removed LNI 192.168.202.108@tcp [ 8876.785221] Key type .llcrypt unregistered [ 8876.787456] Key type ._llcrypt unregistered