[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 465904499 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002343] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009126] LSM: Security Framework initializing [ 0.010046] Yama: becoming mindful. [ 0.011048] SELinux: Initializing. [ 0.012073] *** VALIDATE selinux *** [ 0.019695] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024198] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025128] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027076] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028109] *** VALIDATE tmpfs *** [ 0.029409] *** VALIDATE proc *** [ 0.030223] *** VALIDATE cgroup *** [ 0.031043] *** VALIDATE cgroup2 *** [ 0.033135] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035051] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037025] Spectre V2 : User space: Vulnerable [ 0.038005] Speculative Store Bypass: Vulnerable [ 0.040981] debug: unmapping init [mem 0xffffffffb5459000-0xffffffffb5460fff] [ 0.042959] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043635] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044021] ... version: 2 [ 0.045008] ... bit width: 48 [ 0.046006] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048034] ... max period: 00007fffffffffff [ 0.049017] ... fixed-purpose events: 3 [ 0.050019] ... event mask: 000000070000000f [ 0.052562] rcu: Hierarchical SRCU implementation. [ 0.055673] smp: Bringing up secondary CPUs ... [ 0.056720] x86: Booting SMP configuration: [ 0.057039] .... node #0, CPUs: #1 #2 #3 [ 0.062023] smp: Brought up 1 node, 4 CPUs [ 0.064016] smpboot: Max logical packages: 1 [ 0.065025] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.206000] node 0 deferred pages initialised in 137ms [ 0.213168] devtmpfs: initialized [ 0.214264] x86/mm: Memory block size: 128MB [ 0.219127] gcov: version magic: 0x41383552 [ 0.224247] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.229122] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.234476] pinctrl core: initialized pinctrl subsystem [ 0.236141] [ 0.236702] ************************************************************* [ 0.242049] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.244014] ** ** [ 0.246014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.248012] ** ** [ 0.249015] ** This means that this kernel is built to expose internal ** [ 0.252015] ** IOMMU data structures, which may compromise security on ** [ 0.253011] ** your system. ** [ 0.255012] ** ** [ 0.256010] ** If you see this message and you are not debugging the ** [ 0.258013] ** kernel, report this immediately to your vendor! ** [ 0.259013] ** ** [ 0.261011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262011] ************************************************************* [ 0.264885] NET: Registered protocol family 16 [ 0.266536] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.267060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.269065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.272127] cpuidle: using governor menu [ 0.280134] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.282762] PCI: Using configuration type 1 for base access [ 0.284153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.295088] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.296025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.298071] cryptd: max_cpu_qlen set to 1000 [ 0.300274] ACPI: Added _OSI(Module Device) [ 0.301028] ACPI: Added _OSI(Processor Device) [ 0.303013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.304014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.309916] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.319104] ACPI: Interpreter enabled [ 0.320078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.321014] ACPI: Using IOAPIC for interrupt routing [ 0.322137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.329716] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.339956] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.341043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.343021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.345109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.349735] acpiphp: Slot [2] registered [ 0.350217] acpiphp: Slot [5] registered [ 0.352237] acpiphp: Slot [6] registered [ 0.354175] acpiphp: Slot [7] registered [ 0.355245] acpiphp: Slot [8] registered [ 0.356120] acpiphp: Slot [9] registered [ 0.358178] acpiphp: Slot [10] registered [ 0.360222] acpiphp: Slot [3] registered [ 0.361119] acpiphp: Slot [4] registered [ 0.362220] acpiphp: Slot [11] registered [ 0.363107] acpiphp: Slot [12] registered [ 0.365145] acpiphp: Slot [13] registered [ 0.366096] acpiphp: Slot [14] registered [ 0.367103] acpiphp: Slot [15] registered [ 0.368099] acpiphp: Slot [16] registered [ 0.370154] acpiphp: Slot [17] registered [ 0.371106] acpiphp: Slot [18] registered [ 0.373112] acpiphp: Slot [19] registered [ 0.375197] acpiphp: Slot [20] registered [ 0.377126] acpiphp: Slot [21] registered [ 0.378109] acpiphp: Slot [22] registered [ 0.380175] acpiphp: Slot [23] registered [ 0.381096] acpiphp: Slot [24] registered [ 0.383194] acpiphp: Slot [25] registered [ 0.385266] acpiphp: Slot [26] registered [ 0.387255] acpiphp: Slot [27] registered [ 0.388106] acpiphp: Slot [28] registered [ 0.389118] acpiphp: Slot [29] registered [ 0.391138] acpiphp: Slot [30] registered [ 0.393154] acpiphp: Slot [31] registered [ 0.394207] PCI host bridge to bus 0000:00 [ 0.395024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.398030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.400032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.402033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.406044] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.409033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.411206] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.414243] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.418509] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.432000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.436857] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.439032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.443037] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.446035] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.449405] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.452041] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.454053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.458255] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.463074] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.477023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.484019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.494244] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.502024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.510023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.533025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.552000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.564022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.577021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.609022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.624341] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.635023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.643021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.674021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.683656] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.692023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.700023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.724025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.745777] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.760024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.772070] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.792017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.809011] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.816020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.832022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.854026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.868410] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.875087] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.879398] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.882579] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.886392] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.894492] iommu: Default domain type: Passthrough [ 0.897508] SCSI subsystem initialized [ 0.905059] ACPI: bus type USB registered [ 0.908860] usbcore: registered new interface driver usbfs [ 0.912131] usbcore: registered new interface driver hub [ 0.914211] usbcore: registered new device driver usb [ 0.919944] pps_core: LinuxPPS API ver. 1 registered [ 0.923035] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.930208] PTP clock support registered [ 0.933282] EDAC MC: Ver: 3.0.0 [ 0.936447] PCI: Using ACPI for IRQ routing [ 0.938970] NetLabel: Initializing [ 0.940012] NetLabel: domain hash size = 128 [ 0.941005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.942065] NetLabel: unlabeled traffic allowed by default [ 0.943151] vgaarb: loaded [ 0.945320] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.947016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.955967] clocksource: Switched to clocksource kvm-clock [ 1.111391] VFS: Disk quotas dquot_6.6.0 [ 1.114101] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.131104] *** VALIDATE ramfs *** [ 1.136910] *** VALIDATE hugetlbfs *** [ 1.141551] pnp: PnP ACPI init [ 1.148094] pnp: PnP ACPI: found 6 devices [ 1.172383] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.175077] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.176843] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.178489] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.180190] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.181909] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.184096] NET: Registered protocol family 2 [ 1.186365] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.190559] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.196335] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.207692] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.215313] TCP: Hash tables configured (established 65536 bind 65536) [ 1.221948] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.227802] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.234347] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.240004] NET: Registered protocol family 1 [ 1.246779] RPC: Registered named UNIX socket transport module. [ 1.248785] RPC: Registered udp transport module. [ 1.250897] RPC: Registered tcp transport module. [ 1.252829] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.255442] NET: Registered protocol family 44 [ 1.257093] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.259406] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.261604] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.263778] PCI: CLS 0 bytes, default 64 [ 1.266261] Unpacking initramfs... [ 3.216241] debug: unmapping init [mem 0xffff8b027cc54000-0xffff8b027ffbffff] [ 3.223767] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.228214] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.233229] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 4.217266] Initialise system trusted keyrings [ 4.219626] Key type blacklist registered [ 4.225744] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.240888] zbud: loaded [ 4.244650] *** VALIDATE nfs *** [ 4.246787] *** VALIDATE nfs4 *** [ 4.249296] pstore: using deflate compression [ 4.253731] Platform Keyring initialized [ 4.438216] NET: Registered protocol family 38 [ 4.440330] Key type asymmetric registered [ 4.442325] Asymmetric key parser 'x509' registered [ 4.444659] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.450761] io scheduler mq-deadline registered [ 4.452534] io scheduler kyber registered [ 4.454791] io scheduler bfq registered [ 4.456787] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.459904] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.464606] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.472072] ACPI: Power Button [PWRF] [ 4.483548] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.506572] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.541370] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.556835] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.589520] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.622854] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.658448] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.669420] Non-volatile memory driver v1.3 [ 4.672577] Linux agpgart interface v0.103 [ 4.775752] virtio_blk virtio1: [vda] 146168 512-byte logical blocks (74.8 MB/71.4 MiB) [ 4.786211] vda: detected capacity change from 0 to 74838016 [ 4.829437] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.838419] vdb: detected capacity change from 0 to 1073741824 [ 4.884962] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.892941] vdc: detected capacity change from 0 to 2621440000 [ 4.934448] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.943768] vdd: detected capacity change from 0 to 2621440000 [ 4.969221] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.972322] vde: detected capacity change from 0 to 4294967296 [ 5.023345] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.027249] vdf: detected capacity change from 0 to 4294967296 [ 5.042367] libphy: Fixed MDIO Bus: probed [ 5.055600] usbcore: registered new interface driver usbserial_generic [ 5.060190] usbserial: USB Serial support registered for generic [ 5.063757] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.070901] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.078454] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.088559] mousedev: PS/2 mouse device common for all mice [ 5.097899] rtc_cmos 00:05: RTC can wake from S4 [ 5.109559] rtc_cmos 00:05: registered as rtc0 [ 5.112715] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.117493] intel_pstate: CPU model not supported [ 5.123049] hid: raw HID events driver (C) Jiri Kosina [ 5.125742] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.129106] usbcore: registered new interface driver usbhid [ 5.137602] usbhid: USB HID core driver [ 5.143432] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.147439] drop_monitor: Initializing network drop monitor service [ 5.158327] Initializing XFRM netlink socket [ 5.159590] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.161451] NET: Registered protocol family 10 [ 5.179283] Segment Routing with IPv6 [ 5.182385] NET: Registered protocol family 17 [ 5.185775] mpls_gso: MPLS GSO support [ 5.198072] RAS: Correctable Errors collector initialized. [ 5.201528] AVX version of gcm_enc/dec engaged. [ 5.205815] AES CTR mode by8 optimization enabled [ 5.358311] sched_clock: Marking stable (5358286501, 0)->(6369581748, -1011295247) [ 5.366590] registered taskstats version 1 [ 5.374396] Loading compiled-in X.509 certificates [ 5.379242] zswap: loaded using pool lzo/zbud [ 5.428718] Key type big_key registered [ 5.460433] Key type encrypted registered [ 5.464379] ima: No TPM chip found, activating TPM-bypass! [ 5.469525] ima: Allocated hash algorithm: sha1 [ 5.479581] ima: No architecture policies found [ 5.492254] evm: Initialising EVM extended attributes: [ 5.494316] evm: security.selinux [ 5.495450] evm: security.ima [ 5.508438] evm: security.capability [ 5.515855] evm: HMAC attrs: 0x1 [ 5.529174] rtc_cmos 00:05: setting system clock to 2026-08-07 14:42:58 UTC (1786113778) [ 5.540960] debug: unmapping init [mem 0xffffffffb6403000-0xffffffffb65fffff] [ 5.546314] debug: unmapping init [mem 0xffffffffb5182000-0xffffffffb5458fff] [ 5.556264] Write protecting the kernel read-only data: 28672k [ 5.563426] debug: unmapping init [mem 0xffffffffb3803000-0xffffffffb39fffff] [ 5.570720] debug: unmapping init [mem 0xffffffffb4114000-0xffffffffb41fffff] [ 5.639375] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.652646] systemd[1]: Detected virtualization kvm. [ 5.655515] systemd[1]: Detected architecture x86-64. [ 5.657875] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.696463] systemd[1]: No hostname configured. [ 5.698668] systemd[1]: Set hostname to . [ 5.700935] random: systemd: uninitialized urandom read (16 bytes read) [ 5.703964] systemd[1]: Initializing machine ID from random generator. [ 6.104378] random: systemd: uninitialized urandom read (16 bytes read) [ 6.109996] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.123937] random: systemd: uninitialized urandom read (16 bytes read) [ 6.127920] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 6.139599] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.845584] device-mapper: uevent: version 1.0.3 [ 7.849731] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.909369] random: fast init done [ 10.311629] virtio_net virtio0 ens2: renamed from eth0 [ 10.640023] hrtimer: interrupt took 6618297 ns [ 11.091337] scsi host0: ata_piix [ 11.358130] scsi host1: ata_piix [ 11.359697] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.377294] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.086185] random: crng init done [ 15.089981] random: 7 urandom warning(s) missed due to ratelimiting [ 18.149601] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.378609] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.977641] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.597714] SELinux: Disabled at runtime. [ 23.688968] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.703369] systemd[1]: Detected virtualization kvm. [ 23.705907] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.975865] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.983262] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.001977] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.011983] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.026827] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.038201] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.074180] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ 25.317853] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 27.048278] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.331741] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.487164] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.983303] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.030610] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.279306] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 35.837698] NFS: Registering the id_resolver key type [ 35.839392] Key type id_resolver registered [ 35.845342] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg411-server login: [ 102.760081] libcfs: loading out-of-tree module taints kernel. [ 102.809281] Key type ._llcrypt registered [ 102.811359] Key type .llcrypt registered [ 102.885457] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 115.734590] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 116.560613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 116.569342] alg: No test for adler32 (adler32-zlib) [ 117.675563] Lustre: Lustre: Build Version: 2.17.55_1_gd52575f [ 118.156090] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 119.847230] Key type lgssc registered [ 121.054961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.695025] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 161.462671] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 170.178245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 170.208297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.408066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.424968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 171.474207] Lustre: lustre-MDT0000: new disk, initializing [ 171.535559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.558831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 174.434855] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.225169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.325152] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.351218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 184.357415] Lustre: Skipped 1 previous similar message [ 184.412255] Lustre: lustre-MDT0001: new disk, initializing [ 184.455824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.486892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.500529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 187.398083] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.159318] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.630317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.852708] Lustre: lustre-OST0000: new disk, initializing [ 197.856094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.861267] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.929814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.074733] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.783955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.791453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.877360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 212.328937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.476655] Lustre: lustre-OST0001: new disk, initializing [ 212.479411] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 212.483647] Lustre: 9521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.542170] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.582806] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.640872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 218.650219] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 218.689297] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 226.024539] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 229.210239] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 233.523981] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.551242] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing yml_node [ 241.550860] Lustre: DEBUG MARKER: Client: 2.17.55.1 [ 243.791658] Lustre: DEBUG MARKER: MDS: 2.17.55.1 [ 245.914893] Lustre: DEBUG MARKER: OSS: 2.17.55.1 [ 247.141310] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Fri Aug 7 10:46:58 EDT 2026 [ 259.452524] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 260.991142] Lustre: DEBUG MARKER: === replay-single: start setup 10:47:12 (1786114032) === [ 265.901318] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing check_config_client /mnt/lustre [ 280.412980] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 283.092845] Lustre: 13327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 285.953550] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 290.601562] Lustre: DEBUG MARKER: === replay-single: finish setup 10:47:41 (1786114061) === [ 291.928165] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 10:47:43 (1786114063) [ 296.833879] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 297.987916] Lustre: Failing over lustre-MDT0000 [ 298.131294] Lustre: server umount lustre-MDT0000 complete [ 300.512382] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.519194] Lustre: Skipped 1 previous similar message [ 303.445748] LustreError: 9946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.458631] LustreError: 9946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 305.632663] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 305.645481] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 308.563996] LustreError: 9946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.753649] LustreError: 9946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.762780] LustreError: 9946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 315.874073] LustreError: 6524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 315.886545] LustreError: 6524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 316.899322] Lustre: 3648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786114073/real 1786114073] req@ffff8b01c7fa0000 x1872876159463424/t0(0) o400->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786114089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 316.920642] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 317.086785] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 317.088934] LDISKFS-fs (dm-0): recovery complete [ 317.099990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 323.924192] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.933123] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 326.112339] LustreError: 3645:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b01c7f03100 x1872876159471488/t0(0) o250->MGC192.168.204.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 326.304473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 327.850160] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 328.812270] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 331.755791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 331.805372] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 335.942217] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 337.196630] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 343.685727] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 10:48:35 (1786114115) [ 345.084781] Lustre: Failing over lustre-OST0000 [ 345.170281] Lustre: server umount lustre-OST0000 complete [ 346.089084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 346.097996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.119812] Lustre: Skipped 2 previous similar messages [ 346.128851] LustreError: 8432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.156227] LustreError: 8432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 347.108149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 347.113135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.657365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 360.793544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 361.954764] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 361.976515] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 361.980949] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 361.986733] Lustre: Skipped 3 previous similar messages [ 364.255807] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 370.361298] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 371.427218] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 378.219530] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 10:49:09 (1786114149) [ 383.788103] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 385.420262] Lustre: Failing over lustre-MDT0000 [ 385.675890] Lustre: server umount lustre-MDT0000 complete [ 386.533149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.537200] LustreError: 9946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.552657] Lustre: Skipped 2 previous similar messages [ 386.565182] LustreError: 9946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 402.847094] Lustre: 3649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786114159/real 1786114159] req@ffff8b02d03bea00 x1872876159516032/t0(0) o400->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786114175 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 402.875748] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 404.633963] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 404.636930] LDISKFS-fs (dm-0): recovery complete [ 404.643091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.127785] LustreError: 3645:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b02f71c5500 x1872876159524480/t0(0) o250->MGC192.168.204.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 412.294223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.665814] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 417.525953] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 417.534629] Lustre: lustre-MDT0000: Denying connection for new client 0d515df6-232c-485a-9431-bbac7a6904c1 (at 192.168.204.11@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 417.764839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 417.770125] Lustre: Skipped 1 previous similar message [ 422.743164] Lustre: lustre-MDT0000: Denying connection for new client 0d515df6-232c-485a-9431-bbac7a6904c1 (at 192.168.204.11@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 427.861517] Lustre: lustre-MDT0000: Denying connection for new client 0d515df6-232c-485a-9431-bbac7a6904c1 (at 192.168.204.11@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 432.982165] Lustre: lustre-MDT0000: Denying connection for new client 0d515df6-232c-485a-9431-bbac7a6904c1 (at 192.168.204.11@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 438.101957] Lustre: lustre-MDT0000: Denying connection for new client 0d515df6-232c-485a-9431-bbac7a6904c1 (at 192.168.204.11@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 448.340430] Lustre: lustre-MDT0000: Denying connection for new client 0d515df6-232c-485a-9431-bbac7a6904c1 (at 192.168.204.11@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 448.354866] Lustre: Skipped 1 previous similar message [ 468.823297] Lustre: lustre-MDT0000: Denying connection for new client 0d515df6-232c-485a-9431-bbac7a6904c1 (at 192.168.204.11@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 468.837843] Lustre: Skipped 3 previous similar messages [ 487.500627] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 487.505445] Lustre: 18224:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 8117de1d-0d15-4e75-8fb8-0f68afd8c2f7@ [ 487.516356] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 487.537225] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 487.537867] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 487.545955] Lustre: Skipped 2 previous similar messages [ 487.581677] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 487.589447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 496.801314] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 10:51:08 (1786114268) [ 502.125041] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 503.621390] Lustre: Failing over lustre-MDT0000 [ 503.783396] Lustre: server umount lustre-MDT0000 complete [ 507.872824] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.873266] LustreError: 14841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.885121] Lustre: Skipped 4 previous similar messages [ 507.905184] LustreError: 14841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 29 previous similar messages [ 522.296580] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 522.299379] LDISKFS-fs (dm-0): recovery complete [ 522.305313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.381242] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.519937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.132265] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 526.526965] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 526.531686] Lustre: lustre-MDT0000: Denying connection for new client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 526.544275] Lustre: Skipped 3 previous similar messages [ 527.849629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 593.235909] Lustre: lustre-MDT0000: Denying connection for new client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:03 [ 593.252656] Lustre: Skipped 12 previous similar messages [ 596.500469] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 596.506267] Lustre: 19950:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0d515df6-232c-485a-9431-bbac7a6904c1@ [ 596.515644] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 596.525157] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 596.525827] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 596.538302] Lustre: Skipped 2 previous similar messages [ 596.566471] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 596.566493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 604.986461] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 10:52:56 (1786114376) [ 610.108737] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 611.388984] Lustre: Failing over lustre-MDT0000 [ 611.530284] Lustre: server umount lustre-MDT0000 complete [ 611.812233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 611.821949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.302439] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 629.304868] LDISKFS-fs (dm-0): recovery complete [ 629.310714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.381487] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 629.547786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 629.585312] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 630.121260] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 632.291934] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 634.862746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 634.921812] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 634.960062] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 634.960736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 638.759376] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 639.954713] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 645.864805] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 10:53:37 (1786114417) [ 650.794923] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 652.091412] Lustre: Failing over lustre-MDT0000 [ 652.246628] Lustre: server umount lustre-MDT0000 complete [ 655.328425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.329220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 655.329294] LustreError: 9946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.329304] LustreError: 9946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 655.339869] Lustre: Skipped 5 previous similar messages [ 669.772956] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 669.775561] LDISKFS-fs (dm-0): recovery complete [ 669.783915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.857568] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 670.049297] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 671.059337] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 672.487845] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 675.301501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 675.305446] Lustre: Skipped 3 previous similar messages [ 675.346312] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 675.382459] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 675.382874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 678.657463] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 679.749675] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 685.502275] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 10:54:17 (1786114457) [ 690.234388] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 691.350086] Lustre: Failing over lustre-MDT0000 [ 691.487899] Lustre: server umount lustre-MDT0000 complete [ 695.777037] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.777190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 695.786213] Lustre: Skipped 4 previous similar messages [ 709.149417] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 709.151869] LDISKFS-fs (dm-0): recovery complete [ 709.160832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.218045] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 709.412053] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 712.021576] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 712.075400] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 714.787368] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 714.825749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 714.826245] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:193) [ 718.380836] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 719.409805] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 724.995803] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 10:54:56 (1786114496) [ 729.516449] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 730.669312] Lustre: Failing over lustre-MDT0000 [ 730.842621] Lustre: server umount lustre-MDT0000 complete [ 735.200395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 748.847566] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 748.849845] LDISKFS-fs (dm-0): recovery complete [ 748.857713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.925746] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 749.099624] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 749.936323] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 751.793477] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 754.148548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 754.154107] Lustre: Skipped 7 previous similar messages [ 754.218136] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 754.247165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 754.247277] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 757.922590] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 758.944470] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 764.256745] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 10:55:35 (1786114535) [ 769.324646] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 770.743947] Lustre: Failing over lustre-MDT0000 [ 770.752656] LustreError: 3648:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b02d87c9180 x1872876159734400/t0(0) o103->lustre-MDT0001-osp-MDT0000@0@lo:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'jbd2/dm-0-8.0' uid:0 gid:0 projid:4294967295 [ 770.972078] Lustre: server umount lustre-MDT0000 complete [ 774.623543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 774.624609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.638605] Lustre: Skipped 7 previous similar messages [ 789.596645] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 789.601146] LDISKFS-fs (dm-0): recovery complete [ 789.609252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.710173] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 789.873250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.877630] Lustre: Skipped 3 previous similar messages [ 789.920851] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 792.830813] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 795.179139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 795.179149] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 799.398722] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 800.584090] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 807.260481] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 10:56:18 (1786114578) [ 807.914399] Lustre: *** cfs_fail_loc=13b, val=315*** [ 807.918056] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 807.923913] LustreError: 6521:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b02d109f100 x1872876145869952/t38654705666(0) o35->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:521/0 lens 392/456 e 0 to 0 dl 1786114591 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 814.339371] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 815.673832] Lustre: Failing over lustre-MDT0000 [ 815.831022] Lustre: server umount lustre-MDT0000 complete [ 820.707351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 833.430347] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 833.432403] LDISKFS-fs (dm-0): recovery complete [ 833.438543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 833.675150] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 836.603317] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 836.950855] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 836.958177] Lustre: Skipped 1 previous similar message [ 839.164905] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 839.171965] Lustre: Skipped 1 previous similar message [ 839.181790] Lustre: 6521:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b02f7861f80 x1872876145869952/t38654705666(0) o35->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:553/0 lens 392/456 e 0 to 0 dl 1786114623 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 839.200922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 839.201867] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 842.654376] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 843.641636] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 848.918671] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 10:57:00 (1786114620) [ 853.640577] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 854.851823] Lustre: Failing over lustre-MDT0000 [ 855.029961] Lustre: server umount lustre-MDT0000 complete [ 872.968686] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 872.971198] LDISKFS-fs (dm-0): recovery complete [ 872.976497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.075650] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 873.085176] LustreError: Skipped 1 previous similar message [ 873.240842] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 875.925388] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 878.631505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 878.641721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 882.210426] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 883.090734] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 887.940971] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 10:57:39 (1786114659) [ 892.287448] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 892.797827] Lustre: *** cfs_fail_loc=114, val=0*** [ 894.394903] Lustre: Failing over lustre-MDT0000 [ 894.557398] Lustre: server umount lustre-MDT0000 complete [ 899.041227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 899.045674] LustreError: Skipped 1 previous similar message [ 911.135835] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 911.137579] LDISKFS-fs (dm-0): recovery complete [ 911.143588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.355078] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 913.428282] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 916.449408] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 916.458691] Lustre: Skipped 15 previous similar messages [ 916.506558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 916.508309] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 919.998930] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 920.971140] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 926.146263] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 10:58:17 (1786114697) [ 929.981663] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 930.534049] Lustre: *** cfs_fail_loc=128, val=0*** [ 931.957428] Lustre: Failing over lustre-MDT0000 [ 932.106301] Lustre: server umount lustre-MDT0000 complete [ 934.745085] LustreError: 14841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.753513] LustreError: 14841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 108 previous similar messages [ 936.928185] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 936.935648] Lustre: Skipped 15 previous similar messages [ 949.043607] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 949.045890] LDISKFS-fs (dm-0): recovery complete [ 949.054867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.151348] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 954.383702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 954.383788] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 957.479208] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 958.393374] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 963.428127] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 10:58:55 (1786114735) [ 967.383425] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 968.643883] Lustre: Failing over lustre-MDT0000 [ 968.784324] Lustre: server umount lustre-MDT0000 complete [ 984.842158] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 984.847971] LDISKFS-fs (dm-0): recovery complete [ 984.856785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.076724] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 985.081522] Lustre: Skipped 1 previous similar message [ 985.940228] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 985.943993] Lustre: Skipped 3 previous similar messages [ 987.050304] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 990.290355] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 990.294120] Lustre: Skipped 3 previous similar messages [ 990.312910] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 990.314436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 993.130234] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 993.922090] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 999.269474] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 10:59:30 (1786114770) [ 1003.656667] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1004.771698] Lustre: Failing over lustre-MDT0000 [ 1004.936930] Lustre: server umount lustre-MDT0000 complete [ 1020.863137] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1020.865589] LDISKFS-fs (dm-0): recovery complete [ 1020.871215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.939929] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.945291] LustreError: Skipped 3 previous similar messages [ 1023.104566] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1026.665298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1026.665525] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1029.541850] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1030.422518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1035.387285] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 11:00:07 (1786114807) [ 1039.287533] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1043.872031] Lustre: Failing over lustre-MDT0000 [ 1044.082346] Lustre: server umount lustre-MDT0000 complete [ 1047.007851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1047.011441] LustreError: Skipped 2 previous similar messages [ 1060.232779] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1060.235108] LDISKFS-fs (dm-0): recovery complete [ 1060.239386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.425215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.427978] Lustre: Skipped 6 previous similar messages [ 1062.393316] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1067.126065] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1067.126530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1069.916572] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1070.707212] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1085.499633] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 11:00:57 (1786114857) [ 1089.553666] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1090.791105] Lustre: Failing over lustre-MDT0000 [ 1090.949827] Lustre: server umount lustre-MDT0000 complete [ 1107.258874] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1107.260915] LDISKFS-fs (dm-0): recovery complete [ 1107.265631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.467045] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1112.589981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1112.595618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1115.265254] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1116.057135] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1122.769501] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 11:01:34 (1786114894) [ 1126.569683] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1127.618491] Lustre: Failing over lustre-MDT0000 [ 1127.734800] Lustre: server umount lustre-MDT0000 complete [ 1143.824848] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1143.826875] LDISKFS-fs (dm-0): recovery complete [ 1143.833930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1144.020657] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1144.025970] Lustre: Skipped 3 previous similar messages [ 1146.060949] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1149.450848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1149.450870] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1152.202666] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1153.020440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1157.585289] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 11:02:09 (1786114929) [ 1161.546196] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1162.586476] Lustre: Failing over lustre-MDT0000 [ 1162.722377] Lustre: server umount lustre-MDT0000 complete [ 1178.269471] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1178.272648] LDISKFS-fs (dm-0): recovery complete [ 1178.281211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.390760] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1183.716511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1183.720510] Lustre: Skipped 27 previous similar messages [ 1183.765942] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1183.766388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1186.322630] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1187.090095] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1191.591550] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 11:02:43 (1786114963) [ 1195.081393] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1196.051557] Lustre: Failing over lustre-MDT0000 [ 1196.191157] Lustre: server umount lustre-MDT0000 complete [ 1199.072665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.080989] Lustre: Skipped 27 previous similar messages [ 1211.854723] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1211.856884] LDISKFS-fs (dm-0): recovery complete [ 1211.860949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.743364] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1217.575869] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1217.582581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1220.222141] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1220.908162] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1225.400388] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 11:03:17 (1786114997) [ 1229.079613] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1230.080111] Lustre: Failing over lustre-MDT0000 [ 1230.248977] Lustre: server umount lustre-MDT0000 complete [ 1246.338868] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1246.341139] LDISKFS-fs (dm-0): recovery complete [ 1246.345595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1247.062801] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1247.068313] Lustre: Skipped 6 previous similar messages [ 1248.555963] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1251.835042] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1251.840031] Lustre: Skipped 6 previous similar messages [ 1251.864284] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1251.864352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1254.681725] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1255.412142] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1260.214822] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 11:03:51 (1786115031) [ 1263.908559] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1264.863308] Lustre: Failing over lustre-MDT0000 [ 1265.085415] Lustre: server umount lustre-MDT0000 complete [ 1280.607858] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1280.610112] LDISKFS-fs (dm-0): recovery complete [ 1280.616486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.673294] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1280.676833] LustreError: Skipped 6 previous similar messages [ 1282.470461] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1286.162353] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1286.163023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1288.658665] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1289.374329] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1294.007305] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 11:04:25 (1786115065) [ 1297.882515] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1298.856777] Lustre: Failing over lustre-MDT0000 [ 1298.990305] Lustre: server umount lustre-MDT0000 complete [ 1314.632727] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1314.636798] LDISKFS-fs (dm-0): recovery complete [ 1314.644166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1316.903368] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1319.976952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1319.977026] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1322.624574] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1323.404079] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1328.136995] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 11:04:59 (1786115099) [ 1332.065627] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1333.021407] Lustre: Failing over lustre-MDT0000 [ 1333.207704] Lustre: server umount lustre-MDT0000 complete [ 1335.263679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1335.267134] LustreError: Skipped 6 previous similar messages [ 1349.071553] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1349.074147] LDISKFS-fs (dm-0): recovery complete [ 1349.082316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.087237] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1354.770911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1354.771761] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1357.255751] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1358.004912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1362.242802] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 11:05:34 (1786115134) [ 1365.820562] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1366.734400] Lustre: Failing over lustre-MDT0000 [ 1368.880811] Lustre: server umount lustre-MDT0000 complete [ 1384.263143] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1384.264884] LDISKFS-fs (dm-0): recovery complete [ 1384.268656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.264835] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1389.578925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 1389.578968] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 1392.016934] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1392.772457] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1397.140898] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 11:06:08 (1786115168) [ 1400.441442] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1401.353266] Lustre: Failing over lustre-MDT0000 [ 1401.500410] Lustre: server umount lustre-MDT0000 complete [ 1416.950422] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1416.952336] LDISKFS-fs (dm-0): recovery complete [ 1416.958457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.122390] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1417.127656] Lustre: Skipped 7 previous similar messages [ 1418.791961] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1422.341610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:897) [ 1422.342276] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 1425.013831] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1425.790461] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1429.758951] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 11:06:41 (1786115201) [ 1432.947842] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1433.782752] Lustre: Failing over lustre-MDT0000 [ 1433.972676] Lustre: server umount lustre-MDT0000 complete [ 1447.904638] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1447.911266] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 237 previous similar messages [ 1448.716092] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1448.718177] LDISKFS-fs (dm-0): recovery complete [ 1448.721908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.294953] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1454.088323] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1454.088423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1456.146874] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1456.682182] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1460.331370] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 11:07:12 (1786115232) [ 1463.367056] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1464.141925] Lustre: Failing over lustre-MDT0000 [ 1464.293420] Lustre: server umount lustre-MDT0000 complete [ 1479.105459] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1479.107346] LDISKFS-fs (dm-0): recovery complete [ 1479.112553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.803319] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1484.812396] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 1484.812398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 1487.092465] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1487.734061] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1491.661354] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 11:07:43 (1786115263) [ 1494.731549] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1495.555932] Lustre: Failing over lustre-MDT0000 [ 1495.731222] Lustre: server umount lustre-MDT0000 complete [ 1510.420433] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1510.422180] LDISKFS-fs (dm-0): recovery complete [ 1510.427054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1512.025307] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1516.028577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:993) [ 1516.028579] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 1518.213852] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1518.778974] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1522.422627] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 11:08:14 (1786115294) [ 1525.292070] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1526.129388] Lustre: Failing over lustre-MDT0000 [ 1526.240563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1526.243905] Lustre: Skipped 3 previous similar messages [ 1526.346527] Lustre: server umount lustre-MDT0000 complete [ 1540.438130] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1540.439565] LDISKFS-fs (dm-0): recovery complete [ 1540.442962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.889632] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1545.726493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1545.726672] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1547.532644] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1548.076970] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1551.526992] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 11:08:43 (1786115323) [ 1554.152723] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1554.828246] Lustre: Failing over lustre-MDT0000 [ 1554.977487] Lustre: server umount lustre-MDT0000 complete [ 1569.026594] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1569.028275] LDISKFS-fs (dm-0): recovery complete [ 1569.035453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1570.554381] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1574.393850] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1574.393881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1576.229833] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1576.787122] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1580.303104] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:09:12 (1786115352) [ 1582.960242] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1583.655987] Lustre: Failing over lustre-MDT0000 [ 1583.865085] Lustre: server umount lustre-MDT0000 complete [ 1598.276444] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1598.278541] LDISKFS-fs (dm-0): recovery complete [ 1598.282384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.409124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.412983] Lustre: Skipped 15 previous similar messages [ 1599.876744] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1603.580212] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 1603.580217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 1605.492373] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1606.052726] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1609.476368] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 11:09:41 (1786115381) [ 1611.206180] Lustre: 75188:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting d0401fe9-190c-4a32-b38e-28f17d815520 at adminstrative request [ 1614.713836] Lustre: Failing over lustre-MDT0000 [ 1614.971285] Lustre: server umount lustre-MDT0000 complete [ 1627.941625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.440974] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1633.270732] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 1633.270763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1635.120216] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1635.640663] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1638.047022] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1640.935323] Lustre: DEBUG MARKER: before 3156, after 3156 [ 1642.992101] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 11:10:15 (1786115415) [ 1643.346920] Lustre: 77207:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting d0401fe9-190c-4a32-b38e-28f17d815520 at adminstrative request [ 1647.904683] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:10:19 (1786115419) [ 1650.379430] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1651.081977] Lustre: Failing over lustre-MDT0000 [ 1651.222947] Lustre: server umount lustre-MDT0000 complete [ 1665.585974] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1665.587976] LDISKFS-fs (dm-0): recovery complete [ 1665.593659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.134115] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1671.172567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 1671.172569] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 1673.196339] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1673.723916] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1677.086451] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:10:49 (1786115449) [ 1679.762347] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1680.392206] Lustre: Failing over lustre-MDT0000 [ 1680.567583] Lustre: server umount lustre-MDT0000 complete [ 1694.614104] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1694.615498] LDISKFS-fs (dm-0): recovery complete [ 1694.619855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.992460] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1699.808982] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1699.811725] Lustre: Skipped 63 previous similar messages [ 1699.829913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1699.829919] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1701.533929] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1702.021936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1705.147453] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:11:17 (1786115477) [ 1707.489086] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1708.060945] Lustre: Failing over lustre-MDT0000 [ 1708.241838] Lustre: server umount lustre-MDT0000 complete [ 1722.332718] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1722.335767] LDISKFS-fs (dm-0): recovery complete [ 1722.340132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.933788] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1727.486424] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1727.486424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1729.357250] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1729.921629] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1733.356162] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 11:11:45 (1786115505) [ 1735.923647] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1736.581479] Lustre: Failing over lustre-MDT0000 [ 1736.796747] Lustre: server umount lustre-MDT0000 complete [ 1737.695637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.698950] Lustre: Skipped 66 previous similar messages [ 1751.346067] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1751.347905] LDISKFS-fs (dm-0): recovery complete [ 1751.352412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1753.102971] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1756.662700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1756.662699] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1758.589727] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1759.147372] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1762.654398] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:12:14 (1786115534) [ 1765.457037] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1766.172225] Lustre: Failing over lustre-MDT0000 [ 1766.360875] Lustre: server umount lustre-MDT0000 complete [ 1781.267385] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1781.269383] LDISKFS-fs (dm-0): recovery complete [ 1781.273239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.122655] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1783.126495] Lustre: Skipped 16 previous similar messages [ 1783.228500] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1786.858075] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1786.861504] Lustre: Skipped 16 previous similar messages [ 1786.877407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 1786.877499] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 1788.895373] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1789.448409] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1793.094416] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:12:45 (1786115565) [ 1795.808242] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1796.534298] Lustre: Failing over lustre-MDT0000 [ 1796.734133] Lustre: server umount lustre-MDT0000 complete [ 1810.943602] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1810.944784] LDISKFS-fs (dm-0): recovery complete [ 1810.947157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1810.982822] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1810.985450] LustreError: Skipped 16 previous similar messages [ 1812.335205] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1816.581304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 1816.581304] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 1818.253843] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1818.714535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1822.270320] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:13:14 (1786115594) [ 1824.907144] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1825.557959] Lustre: Failing over lustre-MDT0000 [ 1825.677455] Lustre: server umount lustre-MDT0000 complete [ 1839.770570] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1839.772263] LDISKFS-fs (dm-0): recovery complete [ 1839.775732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.367318] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1845.271489] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 1845.271852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 1847.272224] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1847.885840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1851.335156] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:13:43 (1786115623) [ 1853.893881] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1854.562016] Lustre: Failing over lustre-MDT0000 [ 1854.723093] Lustre: server umount lustre-MDT0000 complete [ 1855.456269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1855.458988] LustreError: Skipped 16 previous similar messages [ 1868.774928] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1868.776315] LDISKFS-fs (dm-0): recovery complete [ 1868.779995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.368193] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1873.913827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 1873.914078] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 1875.841765] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1876.373692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1880.093361] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:14:12 (1786115652) [ 1883.037805] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1883.819912] Lustre: Failing over lustre-MDT0000 [ 1883.977263] Lustre: server umount lustre-MDT0000 complete [ 1898.476539] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1898.478515] LDISKFS-fs (dm-0): recovery complete [ 1898.482177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.095704] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1904.126499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 1904.126536] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 1906.114501] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1906.692484] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1910.328971] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:14:42 (1786115682) [ 1913.394433] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1914.159803] Lustre: Failing over lustre-MDT0000 [ 1914.348562] Lustre: server umount lustre-MDT0000 complete [ 1929.425284] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1929.426507] LDISKFS-fs (dm-0): recovery complete [ 1929.430064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1929.569615] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1929.572085] Lustre: Skipped 16 previous similar messages [ 1931.137564] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1934.840450] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 1934.840463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 1937.029668] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1937.582156] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1941.671487] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 11:15:13 (1786115713) [ 1944.579191] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1945.256561] Lustre: Failing over lustre-MDT0000 [ 1945.419899] Lustre: server umount lustre-MDT0000 complete [ 1960.259792] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1960.261405] LDISKFS-fs (dm-0): recovery complete [ 1960.266264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.077906] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1965.561517] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 1965.561576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 1967.522804] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1968.089304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1971.759258] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 11:15:43 (1786115743) [ 1972.116738] Lustre: 98715:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting d0401fe9-190c-4a32-b38e-28f17d815520 at adminstrative request [ 1975.668865] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 11:15:47 (1786115747) [ 1978.015628] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1978.699413] Lustre: Failing over lustre-MDT0000 [ 1978.937605] Lustre: server umount lustre-MDT0000 complete [ 1983.491913] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1983.493651] LDISKFS-fs (dm-0): recovery complete [ 1983.496687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.610931] Lustre: lustre-MDT0000: Aborting client recovery [ 1983.612876] LustreError: 99995:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1983.616055] Lustre: 100028:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1983.619933] Lustre: 100028:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client d0401fe9-190c-4a32-b38e-28f17d815520@ [ 1983.623451] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1983.626370] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1983.629882] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1983.653089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1505) [ 1983.653093] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 1985.112085] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1989.089183] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1998.024880] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 11:16:10 (1786115770) [ 2000.946355] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2001.695427] Lustre: Failing over lustre-MDT0000 [ 2001.887864] Lustre: server umount lustre-MDT0000 complete [ 2006.694207] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2006.696699] LDISKFS-fs (dm-0): recovery complete [ 2006.701881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2006.870072] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2006.915563] Lustre: lustre-MDT0000: Aborting client recovery [ 2006.917948] LustreError: 101855:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2006.921318] Lustre: 101890:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2006.925514] Lustre: 101890:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 2006.928530] Lustre: 101890:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2006.932820] Lustre: 101890:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2006.935905] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2006.940079] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 2006.947639] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 2006.970911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1537) [ 2006.970936] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1537) [ 2008.472139] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2012.129728] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2017.885219] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2017.886812] Lustre: Skipped 1 previous similar message [ 2020.030941] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 11:16:32 (1786115792) [ 2022.772693] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2023.445952] Lustre: Failing over lustre-MDT0000 [ 2023.535167] Lustre: server umount lustre-MDT0000 complete [ 2028.320755] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2028.322826] LDISKFS-fs (dm-0): recovery complete [ 2028.326941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.471846] Lustre: lustre-MDT0000: Aborting client recovery [ 2028.473777] LustreError: 103721:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2028.476821] Lustre: 103754:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2028.480408] Lustre: 103754:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 2028.483771] Lustre: 103754:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2028.488483] Lustre: 103754:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2028.492103] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2028.496806] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 2028.505135] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 2028.527388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 2028.527392] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 2029.964350] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2033.633659] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2041.878905] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 11:16:53 (1786115813) [ 2042.302295] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2042.304989] LustreError: 70701:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b02faa46d80 x1872876146797568/t201863462916(0) o36->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:246/0 lens 512/456 e 0 to 0 dl 1786115826 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 2045.014327] Lustre: Failing over lustre-MDT0000 [ 2045.108879] Lustre: server umount lustre-MDT0000 complete [ 2048.317647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.339442] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.349650] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 256 previous similar messages [ 2048.495534] Lustre: lustre-MDT0000: Aborting client recovery [ 2048.497304] LustreError: 105042:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2048.500431] Lustre: 105074:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2048.504291] Lustre: 105074:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 2048.507351] Lustre: 105074:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2048.511467] Lustre: 105074:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2048.515149] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2048.518873] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 2048.526110] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 2048.547257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1601) [ 2048.547257] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 2049.900315] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2053.603029] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2061.444051] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 2061.991531] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 11:17:14 (1786115834) [ 2064.426909] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2065.276691] Lustre: Failing over lustre-MDT0000 [ 2065.434869] Lustre: server umount lustre-MDT0000 complete [ 2069.289075] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2069.291087] LDISKFS-fs (dm-0): recovery complete [ 2069.294198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.435365] Lustre: lustre-MDT0000: Aborting client recovery [ 2069.437030] LustreError: 107001:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2069.439597] Lustre: 107033:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2069.442800] Lustre: 107033:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 2069.444968] Lustre: 107033:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2069.447932] Lustre: 107033:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2069.450181] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2069.452687] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 2069.458234] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 2069.477580] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 2069.477585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1633) [ 2070.862533] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2074.592578] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2082.862965] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:17:34 (1786115854) [ 2092.916982] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2093.562380] Lustre: Failing over lustre-MDT0000 [ 2093.725987] Lustre: server umount lustre-MDT0000 complete [ 2107.745260] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2107.746887] LDISKFS-fs (dm-0): recovery complete [ 2107.749644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.169273] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2113.032500] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 2113.032503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 2114.826709] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2115.318485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2123.690816] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:18:15 (1786115895) [ 2130.163320] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2132.979108] Lustre: Failing over lustre-MDT0000 [ 2133.239533] Lustre: server umount lustre-MDT0000 complete [ 2147.553179] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2147.554433] LDISKFS-fs (dm-0): recovery complete [ 2147.558178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.066907] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2153.612224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 2153.612268] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 2155.521830] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2156.042514] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2163.803085] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 11:18:55 (1786115935) [ 2164.499774] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 2164.802253] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2164.804179] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 2166.730803] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 11:18:58 (1786115938) [ 2172.358281] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 2175.253034] Lustre: Failing over lustre-OST0000 [ 2175.305753] Lustre: server umount lustre-OST0000 complete [ 2189.284987] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 2189.286196] LDISKFS-fs (dm-2): recovery complete [ 2189.288848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2191.065605] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2236.865924] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 11:20:08 (1786116008) [ 2239.508275] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2240.398660] Lustre: Failing over lustre-MDT0000 [ 2240.480659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2240.482916] Lustre: Skipped 3 previous similar messages [ 2240.547916] Lustre: server umount lustre-MDT0000 complete [ 2254.486588] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2254.488244] LDISKFS-fs (dm-0): recovery complete [ 2254.491173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.601532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.603527] Lustre: Skipped 20 previous similar messages [ 2255.853349] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2259.962745] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 2259.962922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 2261.678701] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2262.146729] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2275.295222] Lustre: 116346:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786116032/real 1786116032] req@ffff8b02ebe56680 x1872876161340544/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1786116048 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-1-0.0' uid:0 gid:0 projid:4294967295 [ 2275.305124] LustreError: 116346:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2275.305821] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2275.695217] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 11:20:47 (1786116047) [ 2276.320384] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 2277.398667] LustreError: 6518:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2282.463152] LustreError: 6518:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2282.467157] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnecting [ 2283.260322] LustreError: 6519:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2288.607174] LustreError: 6519:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2288.610500] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnecting [ 2289.210372] LustreError: 14841:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2294.239078] LustreError: 14841:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2294.241889] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnecting [ 2294.813643] LustreError: 6518:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2299.871109] LustreError: 6518:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2300.481377] LustreError: 6520:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2305.503127] LustreError: 6520:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2305.506522] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnecting [ 2305.509289] Lustre: Skipped 1 previous similar message [ 2312.286911] LustreError: 14841:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2312.290299] LustreError: 14841:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2317.791101] LustreError: 14841:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2317.793893] LustreError: 14841:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2323.423168] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnecting [ 2323.426694] Lustre: Skipped 2 previous similar messages [ 2330.207262] LustreError: 70701:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2330.211024] LustreError: 70701:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2335.711138] LustreError: 70701:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2335.713760] LustreError: 70701:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2338.481207] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 11:21:50 (1786116110) [ 2339.099627] LustreError: 14841:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2349.395372] Lustre: lustre-MDT0000: Export ffff8b01c6d68000 already connecting from 192.168.204.11@tcp [ 2354.514298] Lustre: lustre-MDT0000: Export ffff8b01c6d68000 already connecting from 192.168.204.11@tcp [ 2359.634323] Lustre: lustre-MDT0000: Export ffff8b01c6d68000 already connecting from 192.168.204.11@tcp [ 2364.755355] Lustre: lustre-MDT0000: Export ffff8b01c6d68000 already connecting from 192.168.204.11@tcp [ 2364.759995] Lustre: Skipped 1 previous similar message [ 2369.874443] Lustre: lustre-MDT0000: Export ffff8b01c6d68000 already connecting from 192.168.204.11@tcp [ 2379.143093] LustreError: 14841:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2379.146076] Lustre: 14841:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b02faa53800 x1872876149427072/t0(0) o38->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:0/0 lens 520/416 e 0 to 0 dl 1786116132 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2380.114576] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnecting [ 2380.118097] Lustre: Skipped 3 previous similar messages [ 2380.119839] LustreError: 70701:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2404.692028] Lustre: lustre-MDT0000: Export ffff8b01c6d68000 already connecting from 192.168.204.11@tcp [ 2404.696221] Lustre: Skipped 1 previous similar message [ 2420.159136] LustreError: 70701:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2420.162873] Lustre: 70701:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b02faa44700 x1872876149432320/t0(0) o38->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:0/0 lens 520/416 e 0 to 0 dl 1786116173 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2422.858491] LustreError: 6520:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2447.698408] Lustre: lustre-MDT0000: Export ffff8b01c6d68000 already connecting from 192.168.204.11@tcp [ 2447.701476] Lustre: Skipped 3 previous similar messages [ 2462.903335] LustreError: 6520:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2462.906083] Lustre: 6520:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b02f5070000 x1872876149435904/t0(0) o38->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:0/0 lens 520/416 e 0 to 0 dl 1786116215 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2463.058225] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnecting [ 2463.061274] Lustre: Skipped 1 previous similar message [ 2463.062836] LustreError: 70701:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2487.634858] Lustre: lustre-MDT0000: Export ffff8b01c6d68000 already connecting from 192.168.204.11@tcp [ 2487.638088] Lustre: Skipped 2 previous similar messages [ 2503.103121] LustreError: 70701:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2503.106221] Lustre: 70701:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b02ebed8380 x1872876149438848/t0(0) o38->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:0/0 lens 520/416 e 0 to 0 dl 1786116256 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2506.827215] LustreError: 6520:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2546.871102] LustreError: 6520:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2546.874257] Lustre: 6520:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b02d87c1c00 x1872876149442432/t0(0) o38->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:0/0 lens 520/416 e 0 to 0 dl 1786116299 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2547.026236] LustreError: 70701:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2548.799127] LustreError: 70701:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout interrupted [ 2550.251486] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 11:25:22 (1786116322) [ 2552.691947] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2553.859202] Lustre: Failing over lustre-MDT0000 [ 2553.946630] Lustre: server umount lustre-MDT0000 complete [ 2556.895899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2556.896091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.898833] LustreError: Skipped 17 previous similar messages [ 2556.903565] Lustre: Skipped 67 previous similar messages [ 2558.079915] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2558.081283] LDISKFS-fs (dm-0): recovery complete [ 2558.084774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.128233] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.132343] LustreError: Skipped 13 previous similar messages [ 2558.191949] Lustre: *** cfs_fail_loc=712, val=0*** [ 2558.193995] LustreError: 43891:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff8b02f7863100 x1872876161484544/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2558.243436] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2558.243514] Lustre: lustre-MDT0000: Aborting client recovery [ 2558.246021] Lustre: Skipped 20 previous similar messages [ 2558.248492] LustreError: 121187:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2558.250531] Lustre: 121219:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2558.254402] Lustre: 121219:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 2558.257639] Lustre: 121219:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2558.261908] Lustre: 121219:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2558.265139] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2558.268691] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2558.275134] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 2558.293484] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 2558.295719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 2559.645430] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2563.552825] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2563.562534] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2563.564532] Lustre: Skipped 78 previous similar messages [ 2570.260702] Lustre: Failing over lustre-MDT0000 [ 2570.496743] Lustre: server umount lustre-MDT0000 complete [ 2583.210278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.598767] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2587.986244] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2587.989512] Lustre: Skipped 10 previous similar messages [ 2588.646169] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2588.650032] Lustre: Skipped 10 previous similar messages [ 2588.664989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 2588.665038] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 2590.399981] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2590.896532] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2594.195427] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 11:26:06 (1786116366) [ 2594.228704] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnecting [ 2594.231331] Lustre: Skipped 2 previous similar messages [ 2597.272866] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 11:26:09 (1786116369) [ 2597.584397] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2597.586190] LustreError: 6528:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b01ca013b80 x1872876149498368/t0(0) o700->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:46/0 lens 264/248 e 0 to 0 dl 1786116381 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 2615.175805] Lustre: Failing over lustre-MDT0000 [ 2615.384593] Lustre: server umount lustre-MDT0000 complete [ 2628.168947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.513202] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2633.712307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 2633.712316] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 2635.447434] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2635.928186] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2639.986620] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 11:26:51 (1786116411) [ 2640.806883] Lustre: Failing over lustre-OST0000 [ 2640.855808] Lustre: server umount lustre-OST0000 complete [ 2649.056213] LustreError: 17104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.061694] LustreError: 17104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 86 previous similar messages [ 2653.480538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.291837] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2658.886375] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2659.419960] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2724.664608] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 11:28:16 (1786116496) [ 2727.109615] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2727.813888] Lustre: Failing over lustre-MDT0000 [ 2727.953162] Lustre: server umount lustre-MDT0000 complete [ 2741.610051] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2741.611395] LDISKFS-fs (dm-0): recovery complete [ 2741.613834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.933916] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2746.924038] Lustre: *** cfs_fail_loc=216, val=0*** [ 2746.924046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 2746.930223] LustreError: 127335:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2747.935401] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 2809.327324] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 11:29:41 (1786116581) [ 2817.119679] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 11:29:49 (1786116589) [ 2818.011370] Lustre: Failing over lustre-MDT0000 [ 2818.135451] Lustre: server umount lustre-MDT0000 complete [ 2830.657381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.039365] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2835.941465] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2835.942982] LustreError: 128913:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b02ebef2300 x1872876149616128/t0(0) o101->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:327/0 lens 328/344 e 0 to 0 dl 1786116662 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2894.162363] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnected, waiting for 2 clients in recovery for 1:04 [ 2894.168393] Lustre: 128913:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2894.190089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 2894.190132] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 2896.118494] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2896.614419] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2900.308096] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 11:31:12 (1786116672) [ 2901.687917] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2904.390036] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2905.018561] Lustre: Failing over lustre-MDT0000 [ 2905.269725] Lustre: server umount lustre-MDT0000 complete [ 2919.450709] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2919.453107] LDISKFS-fs (dm-0): recovery complete [ 2919.457348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2919.567673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2919.571579] Lustre: Skipped 6 previous similar messages [ 2920.911684] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2925.044220] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3105) [ 2925.044231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3137) [ 2926.648611] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2927.121903] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2930.285977] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 11:31:42 (1786116702) [ 2930.632699] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2934.315675] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2934.938706] Lustre: Failing over lustre-MDT0000 [ 2935.021577] Lustre: server umount lustre-MDT0000 complete [ 2949.177051] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2949.178379] LDISKFS-fs (dm-0): recovery complete [ 2949.180799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.668742] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2954.741755] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3137) [ 2954.741791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 2956.588062] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2957.119802] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2960.693782] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 11:32:12 (1786116732) [ 2961.067162] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2964.710220] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2965.330089] Lustre: Failing over lustre-MDT0000 [ 2965.410033] Lustre: server umount lustre-MDT0000 complete [ 2979.303285] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2979.304684] LDISKFS-fs (dm-0): recovery complete [ 2979.307711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2980.722875] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2984.441673] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3169) [ 2984.441685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201) [ 2988.804221] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 11:32:40 (1786116760) [ 2990.157679] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2990.160035] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2990.162438] LustreError: 6521:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b02facc2d80 x1872876149666944/t257698037777(0) o35->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:439/0 lens 392/456 e 0 to 0 dl 1786116774 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2991.090321] Lustre: Failing over lustre-MDT0000 [ 2991.262937] Lustre: server umount lustre-MDT0000 complete [ 3004.083491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.596358] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3009.524595] Lustre: 6522:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b02f5064380 x1872876149666944/t257698037777(0) o35->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:458/0 lens 392/456 e 0 to 0 dl 1786116793 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3009.531358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 3009.531522] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3201) [ 3011.351325] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3011.891327] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3015.241710] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 11:33:07 (1786116787) [ 3015.571730] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3015.573219] LustreError: 9946:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b02ebeda300 x1872876149680128/t261993005072(0) o36->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:508/0 lens 504/448 e 0 to 0 dl 1786116843 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3019.214253] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3019.837545] Lustre: Failing over lustre-MDT0000 [ 3019.934324] Lustre: server umount lustre-MDT0000 complete [ 3033.919844] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3033.921248] LDISKFS-fs (dm-0): recovery complete [ 3033.924305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3035.391735] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3039.215853] Lustre: 9946:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b02faa51500 x1872876149680128/t261993005072(0) o36->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:532/0 lens 504/2880 e 0 to 0 dl 1786116867 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3039.220567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3265) [ 3039.220568] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 3040.891728] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3041.388576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3044.567876] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 11:33:36 (1786116816) [ 3044.916762] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3044.919101] LustreError: 70701:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b02fad03800 x1872876149693952/t266287972368(0) o36->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:537/0 lens 504/448 e 0 to 0 dl 1786116872 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3046.187877] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3048.365230] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3048.965791] Lustre: Failing over lustre-MDT0000 [ 3049.047517] Lustre: server umount lustre-MDT0000 complete [ 3063.106436] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3063.107792] LDISKFS-fs (dm-0): recovery complete [ 3063.110163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3064.590507] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3068.395374] Lustre: 6519:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b02ebee9500 x1872876149693952/t266287972368(0) o36->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:561/0 lens 504/2880 e 0 to 0 dl 1786116896 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3068.401741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3297) [ 3068.401879] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 3068.402745] Lustre: 6519:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 3072.696111] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 11:34:04 (1786116844) [ 3073.047313] LustreError: 14841:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b02d02dca80 x1872876149706112/t270582939664(0) o36->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:566/0 lens 504/448 e 0 to 0 dl 1786116901 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3073.057443] LustreError: 14841:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 1 previous similar message [ 3074.341370] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3074.343013] Lustre: Skipped 1 previous similar message [ 3077.110824] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3077.757317] Lustre: Failing over lustre-MDT0000 [ 3077.850064] Lustre: server umount lustre-MDT0000 complete [ 3091.717419] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3091.718670] LDISKFS-fs (dm-0): recovery complete [ 3091.721097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3093.134697] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3097.070821] Lustre: 14841:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b02f7128700 x1872876149706112/t270582939664(0) o36->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:590/0 lens 504/2880 e 0 to 0 dl 1786116925 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3097.078114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3329) [ 3097.078263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 3101.050779] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 11:34:33 (1786116873) [ 3101.389422] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3102.675165] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3102.676873] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 3102.678194] Lustre: Skipped 2 previous similar messages [ 3106.091879] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3106.728970] Lustre: Failing over lustre-MDT0000 [ 3106.814296] Lustre: server umount lustre-MDT0000 complete [ 3120.764105] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3120.765188] LDISKFS-fs (dm-0): recovery complete [ 3120.767448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.128739] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3126.253134] Lustre: 6522:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b02ec2c1500 x1872876149718528/t274877906960(0) o35->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:575/0 lens 392/456 e 0 to 0 dl 1786116910 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3126.258423] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329) [ 3126.258432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 3130.572535] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 11:35:02 (1786116902) [ 3130.866271] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 3130.868533] LustreError: 6519:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b01ca173480 x1872876149728384/t279172874255(0) o101->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:623/0 lens 664/608 e 0 to 0 dl 1786116958 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 3130.875950] LustreError: 6519:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 1 previous similar message [ 3191.128619] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnecting [ 3191.131396] Lustre: Skipped 4 previous similar messages [ 3191.139423] Lustre: 6519:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b02fa947100 x1872876149728384/t279172874255(0) o101->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:684/0 lens 664/3488 e 0 to 0 dl 1786117019 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 3193.306433] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 11:36:05 (1786116965) [ 3195.843864] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3196.491849] Lustre: Failing over lustre-MDT0000 [ 3196.571923] Lustre: server umount lustre-MDT0000 complete [ 3197.919828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.919894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3197.924039] Lustre: Skipped 56 previous similar messages [ 3197.926201] LustreError: Skipped 12 previous similar messages [ 3210.660605] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3210.661805] LDISKFS-fs (dm-0): recovery complete [ 3210.664814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.701204] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3210.704805] LustreError: Skipped 12 previous similar messages [ 3210.813538] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3210.817530] Lustre: Skipped 15 previous similar messages [ 3211.602627] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3211.606435] Lustre: Skipped 12 previous similar messages [ 3212.210632] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3215.841268] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3215.843284] Lustre: Skipped 52 previous similar messages [ 3215.850971] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3215.853851] Lustre: Skipped 12 previous similar messages [ 3215.868290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 3215.868366] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3361) [ 3217.529943] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3218.004488] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3231.703246] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 11:36:43 (1786117003) [ 3234.609830] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3235.280197] Lustre: Failing over lustre-MDT0000 [ 3235.383922] Lustre: server umount lustre-MDT0000 complete [ 3249.303100] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3249.304973] LDISKFS-fs (dm-0): recovery complete [ 3249.308317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.789271] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3254.773501] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3393) [ 3254.773506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 3256.443735] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3256.895230] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3259.040357] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3263.965052] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 11:37:16 (1786117036) [ 3273.972789] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3274.609456] Lustre: Failing over lustre-MDT0000 [ 3274.979164] Lustre: server umount lustre-MDT0000 complete [ 3275.232898] LustreError: 119438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.240187] LustreError: 119438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 169 previous similar messages [ 3289.137331] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3289.138696] LDISKFS-fs (dm-0): recovery complete [ 3289.141321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3290.563300] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3294.794498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 3294.794539] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 3296.539731] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3297.038985] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3310.663950] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 11:38:02 (1786117082) [ 3314.902468] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3315.545287] Lustre: Failing over lustre-MDT0000 [ 3315.693395] Lustre: server umount lustre-MDT0000 complete [ 3329.784081] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3329.786318] LDISKFS-fs (dm-0): recovery complete [ 3329.790325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3331.320452] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3335.167990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 3335.167990] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 3336.994690] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3337.528508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3340.971245] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 3341.468640] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3343.600173] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 11:38:35 (1786117115) [ 3349.453414] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3410.056898] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3410.058346] Lustre: Skipped 1 previous similar message [ 3410.059519] LustreError: 9946:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b02faa45f80 x1872876152517504/t296352743435(0) o36->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:148/0 lens 66040/440 e 0 to 0 dl 1786117238 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3469.652985] Lustre: 119438:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b02d0566c50 x1872876152517504/t296352743435(0) o36->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:207/0 lens 66040/440 e 0 to 0 dl 1786117297 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3472.613092] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3473.126357] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 11:40:45 (1786117245) [ 3477.114285] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3477.927413] Lustre: Failing over lustre-MDT0000 [ 3478.129079] Lustre: server umount lustre-MDT0000 complete [ 3492.076084] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3492.077479] LDISKFS-fs (dm-0): recovery complete [ 3492.080898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.477399] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3497.641318] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833) [ 3497.641344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865) [ 3499.489994] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3500.013213] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3503.818460] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 11:41:15 (1786117275) [ 3509.254147] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3512.326750] Lustre: Failing over lustre-OST0000 [ 3512.366578] Lustre: server umount lustre-OST0000 complete [ 3526.390673] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3526.392852] LDISKFS-fs (dm-2): recovery complete [ 3526.396948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3526.443625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.446382] Lustre: Skipped 12 previous similar messages [ 3528.257235] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3539.505780] Lustre: Failing over lustre-OST0000 [ 3539.542677] Lustre: server umount lustre-OST0000 complete [ 3552.337511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3554.382468] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3558.157880] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3558.730776] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3592.694797] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 11:42:44 (1786117364) [ 3593.590030] Lustre: Failing over lustre-MDT0000 [ 3593.697268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3593.831429] Lustre: server umount lustre-MDT0000 complete [ 3606.428874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.823502] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3611.634714] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865) [ 3611.634719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897) [ 3618.992725] Lustre: Failing over lustre-MDT0000 [ 3619.174702] Lustre: server umount lustre-MDT0000 complete [ 3631.807660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.235229] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3637.235223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929) [ 3637.235244] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897) [ 3639.003329] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3639.498415] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3642.831278] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 11:43:34 (1786117414) [ 3653.974760] Lustre: Failing over lustre-OST0000 [ 3654.024975] Lustre: server umount lustre-OST0000 complete [ 3666.649196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3668.498558] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3672.010727] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3672.474387] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3676.086684] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 11:44:08 (1786117448) [ 3676.729558] Lustre: Failing over lustre-MDT0000 [ 3676.983911] Lustre: server umount lustre-MDT0000 complete [ 3680.222326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.256222] Lustre: *** cfs_fail_loc=605, val=0*** [ 3680.257405] LustreError: 164206:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc09d7100 failed: rc = -95 [ 3680.260298] LustreError: 164206:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 3680.262120] LustreError: 164206:0:(obd_mount.c:259:lustre_start_simple()) MGS setup error -95 [ 3680.263971] LustreError: 164206:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 3680.265839] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3680.267492] LustreError: 164206:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3680.271833] Lustre: server umount lustre-MDT0000 complete [ 3680.273092] LustreError: 164206:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3682.942903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.460590] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3687.032798] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 11:44:19 (1786117459) [ 3688.437926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 3688.437929] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 3690.766421] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3691.688412] Lustre: Failing over lustre-MDT0000 [ 3691.914735] Lustre: server umount lustre-MDT0000 complete [ 3705.806747] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3705.808749] LDISKFS-fs (dm-0): recovery complete [ 3705.813281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.160865] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3710.291395] Lustre: *** cfs_fail_loc=707, val=0*** [ 3769.690456] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 3769.702457] Lustre: 166375:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3769.705813] Lustre: 166375:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 1 previous similar message [ 3769.787273] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 3769.787279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 3771.669898] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3772.149740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3775.821133] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 11:45:47 (1786117547) [ 3799.199389] LustreError: 6518:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b01ca013100 x1872876153397248/t0(0) o101->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:493/0 lens 664/0 e 0 to 0 dl 1786117583 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3799.205819] LustreError: 6518:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3810.231098] LustreError: 6518:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3810.236408] LustreError: 39702:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b01c8482a00 x1872876153398528/t0(0) o35->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:504/0 lens 392/0 e 0 to 0 dl 1786117594 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3815.904630] LustreError: 43904:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b02ebed9f80 x1872876163095552/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:509/0 lens 544/0 e 0 to 0 dl 1786117599 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3815.910796] LustreError: 43904:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 48 previous similar messages [ 3822.092151] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 11:46:34 (1786117594) [ 3845.472447] LustreError: 42674:0:(tgt_handler.c:2836:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 3856.503141] LustreError: 42674:0:(tgt_handler.c:2836:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 3859.448975] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 11:47:11 (1786117631) [ 3882.467105] LustreError: 6520:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b02f712a300 x1872876153421312/t0(0) o101->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:576/0 lens 576/0 e 0 to 0 dl 1786117666 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3882.473318] LustreError: 6520:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3887.575112] LustreError: 6520:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3887.579392] LustreError: 70701:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b02ebed9880 x1872876153422080/t0(0) o101->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:581/0 lens 664/0 e 0 to 0 dl 1786117671 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3897.824089] LustreError: 43887:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b02ebee8e00 x1872876163135488/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:591/0 lens 544/0 e 0 to 0 dl 1786117681 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 3897.830416] LustreError: 43887:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 118 previous similar messages [ 3898.383070] LustreError: 6520:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3910.272522] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 11:48:02 (1786117682) [ 4007.506448] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 11:49:39 (1786117779) [ 4032.328328] LustreError: 119438:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b02ebef1f80 x1872876153500032/t0(0) o101->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:726/0 lens 576/0 e 0 to 0 dl 1786117816 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4032.338559] LustreError: 119438:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 99 previous similar messages [ 4032.342074] LustreError: 119438:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 4032.346605] LustreError: 119438:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 4032.767068] LustreError: 119438:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4048.584936] LustreError: 119438:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 4048.588249] LustreError: 119438:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 4049.007108] LustreError: 119438:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4049.009728] LustreError: 119438:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 4064.416996] LustreError: 70701:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b02fa909f80 x1872876153520000/t0(0) o36->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:3/0 lens 504/0 e 0 to 0 dl 1786117848 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4064.423734] LustreError: 70701:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 80 previous similar messages [ 4076.509322] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 11:50:48 (1786117848) [ 4101.002554] Lustre: DEBUG MARKER: phase 2 [ 4104.046900] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 11:51:16 (1786117876) [ 4173.858585] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 11:52:25 (1786117945) [ 4174.337372] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 4174.869984] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 11:52:26 (1786117946) [ 4176.606867] Lustre: DEBUG MARKER: Started rundbench load pid=134903 ... [ 4180.131639] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4181.659086] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 4182.389437] Lustre: Failing over lustre-MDT0000 [ 4182.729633] Lustre: server umount lustre-MDT0000 complete [ 4183.457081] LustreError: 14841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.463047] LustreError: 14841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 109 previous similar messages [ 4184.544414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4184.547387] LustreError: Skipped 7 previous similar messages [ 4184.549011] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.553069] Lustre: Skipped 38 previous similar messages [ 4196.815974] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4196.817555] LDISKFS-fs (dm-0): recovery complete [ 4196.820608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4196.853703] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4196.858093] LustreError: Skipped 8 previous similar messages [ 4196.923279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.926427] Lustre: Skipped 6 previous similar messages [ 4196.937097] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4196.940934] Lustre: Skipped 11 previous similar messages [ 4198.294356] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4201.810840] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4201.814243] Lustre: Skipped 11 previous similar messages [ 4201.958240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4201.961487] Lustre: Skipped 41 previous similar messages [ 4201.965838] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4201.968260] Lustre: Skipped 11 previous similar messages [ 4201.985294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 4201.986893] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 4203.908842] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4204.480178] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4209.490456] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4210.993403] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 4211.643278] Lustre: Failing over lustre-MDT0001 [ 4211.647920] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [ 4211.651531] Lustre: Skipped 3 previous similar messages [ 4211.850066] Lustre: server umount lustre-MDT0001 complete [ 4226.110105] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 4226.112094] LDISKFS-fs (dm-1): recovery complete [ 4226.118228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.544816] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4232.033713] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:803 to 0x2c0000400:833) [ 4232.033738] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:820 to 0x280000400:865) [ 4233.949023] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4234.504709] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4239.598382] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4241.180348] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 4241.860047] Lustre: Failing over lustre-MDT0000 [ 4241.888471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4242.061220] Lustre: server umount lustre-MDT0000 complete [ 4256.452966] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4256.455500] LDISKFS-fs (dm-0): recovery complete [ 4256.459279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4258.089957] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4261.887481] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 4261.887523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 4263.844817] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4264.380325] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4269.408562] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4270.951657] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 4271.632112] Lustre: Failing over lustre-MDT0001 [ 4271.891661] Lustre: server umount lustre-MDT0001 complete [ 4285.878478] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 4285.880748] LDISKFS-fs (dm-1): recovery complete [ 4285.885252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.363912] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4291.217097] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1224 to 0x2c0000400:1249) [ 4291.217103] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1255 to 0x280000400:1281) [ 4293.041842] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4293.584259] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4322.358129] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 11:54:54 (1786118094) [ 4445.316093] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4456.058970] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 4456.724743] Lustre: Failing over lustre-MDT0001 [ 4456.782387] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [ 4456.784390] Lustre: Skipped 3 previous similar messages [ 4460.000950] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4460.002902] Lustre: Skipped 2 previous similar messages [ 4463.191323] Lustre: server umount lustre-MDT0001 complete [ 4479.507559] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4479.509785] LDISKFS-fs (dm-1): recovery complete [ 4479.519259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.748390] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4492.731781] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2882 to 0x280000400:2913) [ 4492.733243] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2849 to 0x2c0000400:2881) [ 4496.719449] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4498.018164] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4622.618046] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4633.406449] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 4634.070144] Lustre: Failing over lustre-MDT0001 [ 4634.076532] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [ 4634.080891] Lustre: Skipped 1 previous similar message [ 4639.817118] Lustre: server umount lustre-MDT0001 complete [ 4653.983066] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4653.985013] LDISKFS-fs (dm-1): recovery complete [ 4653.989141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4655.416659] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4661.109983] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4078 to 0x2c0000400:4097) [ 4661.110266] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4110 to 0x280000400:4129) [ 4663.003964] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4663.571666] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4678.547490] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 12:00:50 (1786118450) [ 4801.568928] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4812.329312] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 4812.957904] Lustre: Failing over lustre-MDT0001 [ 4812.976623] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 4812.979392] LustreError: Skipped 4 previous similar messages [ 4812.980931] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.986303] Lustre: Skipped 19 previous similar messages [ 4812.988712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4812.990773] Lustre: Skipped 5 previous similar messages [ 4819.005114] Lustre: server umount lustre-MDT0001 complete [ 4823.007661] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.012304] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 72 previous similar messages [ 4833.289608] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4833.291742] LDISKFS-fs (dm-1): recovery complete [ 4833.298324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4833.409275] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4833.420836] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 4833.424152] Lustre: Skipped 5 previous similar messages [ 4833.618630] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4833.623338] Lustre: Skipped 5 previous similar messages [ 4834.760920] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4838.881524] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4838.883272] Lustre: Skipped 19 previous similar messages [ 4839.743256] Lustre: lustre-MDT0001: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4839.745997] Lustre: Skipped 5 previous similar messages [ 4839.762596] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4491 to 0x280000400:4513) [ 4839.764242] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4460 to 0x2c0000400:4481) [ 4841.619229] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4842.225573] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4966.402990] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4977.185922] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 4977.837772] Lustre: Failing over lustre-MDT0000 [ 4977.845050] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 4977.847485] Lustre: Skipped 4 previous similar messages [ 4978.108718] Lustre: server umount lustre-MDT0000 complete [ 4980.193477] LustreError: 151636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786118753 with bad export cookie 4349694307262389842 [ 4980.196526] LustreError: 151636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4992.051469] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4992.052767] LDISKFS-fs (dm-0): recovery complete [ 4992.055322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.092270] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4992.097131] LustreError: Skipped 1 previous similar message [ 4992.162143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4992.164847] Lustre: Skipped 5 previous similar messages [ 4993.481304] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4998.387833] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8447 to 0x2c0000401:8481) [ 4998.387908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8479 to 0x280000401:8513) [ 5000.166798] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5000.723692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5004.205790] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 12:06:16 (1786118776) [ 5127.300351] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5138.035391] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 5138.686977] Lustre: Failing over lustre-MDT0000 [ 5139.016264] Lustre: server umount lustre-MDT0000 complete [ 5153.119053] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5153.120360] LDISKFS-fs (dm-0): recovery complete [ 5153.123268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5154.547429] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5161.959358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10429 to 0x280000401:10465) [ 5162.003294] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10396 to 0x2c0000401:10433) [ 5163.903601] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5164.509631] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5289.650981] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5300.661337] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 5301.516167] Lustre: Failing over lustre-MDT0000 [ 5301.558768] LustreError: 3648:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b01d1a45f80 x1872876192654336/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 5301.728815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5301.731389] Lustre: Skipped 3 previous similar messages [ 5301.789440] Lustre: server umount lustre-MDT0000 complete [ 5316.370098] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5316.371776] LDISKFS-fs (dm-0): recovery complete [ 5316.375946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5317.999311] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5323.877324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11637 to 0x280000401:11681) [ 5323.877506] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11604 to 0x2c0000401:11649) [ 5325.802738] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5326.403180] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5330.234691] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 12:11:42 (1786119102) [ 5336.453201] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5338.057194] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 5338.872382] Lustre: Failing over lustre-OST0000 [ 5338.930828] Lustre: server umount lustre-OST0000 complete [ 5354.001170] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5354.002885] LDISKFS-fs (dm-2): recovery complete [ 5354.008288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5356.722774] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5361.773836] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5362.592397] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5372.221205] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5373.824643] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5374.622223] Lustre: Failing over lustre-OST0000 [ 5374.667286] Lustre: server umount lustre-OST0000 complete [ 5391.035701] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5391.037601] LDISKFS-fs (dm-2): recovery complete [ 5391.044984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5394.212053] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5399.995260] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5400.975684] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5407.852298] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 12:12:59 (1786119179) [ 5532.555791] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5536.655991] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5547.909615] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 5549.081650] Lustre: Failing over lustre-MDT0001 [ 5549.082491] LustreError: lustre-MDT0001-osp-MDT0000: operation ldlm_enqueue to node 0@lo failed: rc = -19 [ 5549.086635] LustreError: Skipped 2 previous similar messages [ 5549.089173] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.094707] Lustre: Skipped 18 previous similar messages [ 5549.097706] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5549.300647] Lustre: server umount lustre-MDT0001 complete [ 5550.048903] LustreError: 52396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5550.061264] LustreError: 52396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 69 previous similar messages [ 5551.441152] Lustre: Failing over lustre-MDT0000 [ 5551.689274] Lustre: server umount lustre-MDT0000 complete [ 5569.045941] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5569.047510] LDISKFS-fs (dm-0): recovery complete [ 5569.054436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.259627] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5569.261845] LDISKFS-fs (dm-1): recovery complete [ 5569.275931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.443649] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5569.451088] Lustre: Skipped 5 previous similar messages [ 5570.661901] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5570.665740] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5570.666411] Lustre: Skipped 6 previous similar messages [ 5570.669789] Lustre: Skipped 18 previous similar messages [ 5570.917990] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5570.927847] Lustre: Skipped 5 previous similar messages [ 5570.950196] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4874 to 0x2c0000400:4897) [ 5570.955137] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4907 to 0x280000400:4929) [ 5572.074626] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5572.221738] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5577.055466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11740 to 0x280000401:11777) [ 5577.056301] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11708 to 0x2c0000401:11745) [ 5579.742906] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5580.520601] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5581.240511] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5706.245059] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5709.347428] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5720.236184] Lustre: DEBUG MARKER: fail mds1 mds2 2 times [ 5720.997652] Lustre: Failing over lustre-MDT0000 [ 5721.171642] Lustre: server umount lustre-MDT0000 complete [ 5722.015101] Lustre: 3649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786119333/real 1786119333] req@ffff8b01c6821880 x1872876197762560/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786119349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5722.025057] Lustre: 3649:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 5722.605831] LustreError: 151641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786119495 with bad export cookie 4349694307265376084 [ 5722.607185] Lustre: Failing over lustre-MDT0001 [ 5722.607777] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5722.607782] LustreError: Skipped 3 previous similar messages [ 5722.610593] LustreError: 151641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5722.792523] Lustre: server umount lustre-MDT0001 complete [ 5738.375306] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5738.379024] LDISKFS-fs (dm-0): recovery complete [ 5738.383898] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5738.385490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5738.387425] LDISKFS-fs (dm-1): recovery complete [ 5738.401846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5738.511148] LustreError: 201433:0:(llog.c:1655:llog_backup()) MGC192.168.204.111@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5738.515524] Lustre: 201433:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.111@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5741.023238] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786119497/real 1786119497] req@ffff8b01cc314a80 x1872876204050304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786119513 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5741.033838] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5743.071248] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786119500/real 1786119500] req@ffff8b01d153dc00 x1872876204050688/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786119516 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5743.080126] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5747.167569] LustreError: 3645:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b01d153e680 x1872876204053632/t0(0) o250->MGC192.168.204.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5747.255875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5747.259329] Lustre: Skipped 6 previous similar messages [ 5748.319255] Lustre: 3649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786119505/real 1786119505] req@ffff8b02f54db100 x1872876204051072/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786119521 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5748.988761] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5749.050242] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5753.465713] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4874 to 0x2c0000400:4929) [ 5753.465723] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4907 to 0x280000400:4961) [ 5753.807176] Lustre: 3648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786119510/real 1786119510] req@ffff8b01cc315c00 x1872876204051584/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786119526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5753.819912] Lustre: 3648:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5759.186591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11708 to 0x2c0000401:11777) [ 5759.186973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11740 to 0x280000401:11809) [ 5761.723124] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5762.468493] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5763.163465] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5765.599173] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786119510/real 1786119510] req@ffff8b01cc314700 x1872876204051840/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786119526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5767.390117] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 12:18:59 (1786119539) [ 5770.643413] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5771.737083] Lustre: Failing over lustre-MDT0000 [ 5771.941873] Lustre: server umount lustre-MDT0000 complete [ 5786.598732] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5786.600514] LDISKFS-fs (dm-0): recovery complete [ 5786.607513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.324853] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5790.039152] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5806.428441] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 5806.444541] Lustre: 204256:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b01d378b800 x1872876235010688/t339302416980(339302416980) o101->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:235/0 lens 520/3488 e 0 to 0 dl 1786119590 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5806.494531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11740 to 0x280000401:11841) [ 5806.494621] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11779 to 0x2c0000401:11809) [ 5808.809252] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5809.462518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5812.976626] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 12:19:44 (1786119584) [ 5815.830855] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5816.893468] Lustre: Failing over lustre-MDT0000 [ 5817.110315] Lustre: server umount lustre-MDT0000 complete [ 5832.052431] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5832.054262] LDISKFS-fs (dm-0): recovery complete [ 5832.060744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.875707] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5836.126044] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5836.128164] LustreError: 201446:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b01d15cea00 x1872876235010688/t339302416980(339302416980) o101->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:265/0 lens 520/664 e 0 to 0 dl 1786119620 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5852.508881] Lustre: lustre-MDT0000: Client d0401fe9-190c-4a32-b38e-28f17d815520 (at 192.168.204.11@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5852.517100] Lustre: 201463:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b01d15cfb80 x1872876235010688/t339302416980(339302416980) o101->d0401fe9-190c-4a32-b38e-28f17d815520@192.168.204.11@tcp:281/0 lens 520/3488 e 0 to 0 dl 1786119636 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5852.555773] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11779 to 0x2c0000401:11841) [ 5852.555893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:11873) [ 5854.932331] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5855.531708] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5859.308254] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 12:20:31 (1786119631) [ 5861.135718] Lustre: Failing over lustre-OST0000 [ 5861.198248] Lustre: server umount lustre-OST0000 complete [ 5862.939954] Lustre: Failing over lustre-MDT0000 [ 5863.071559] Lustre: server umount lustre-MDT0000 complete [ 5876.691884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.548319] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5879.546736] Lustre: lustre-MDT0000: Denying connection for new client ee5672e4-484b-4943-9f37-1d3e916c3f0c (at 192.168.204.11@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5882.357631] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11779 to 0x2c0000401:11873) [ 5886.849018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5889.070719] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5892.081731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:11905) [ 5893.739614] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 12:21:05 (1786119665) [ 5894.133278] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5894.135387] LustreError: 202223:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b02f625d880 x1872876205081600/t360777252871(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:323/0 lens 1312/4320 e 0 to 0 dl 1786119678 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5897.134385] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5897.906707] Lustre: Failing over lustre-MDT0000 [ 5898.208300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5898.210522] Lustre: Skipped 4 previous similar messages [ 5898.272484] Lustre: server umount lustre-MDT0000 complete [ 5910.497538] Lustre: 201468:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786119667/real 1786119667] req@ffff8b02f625ea00 x1872876205081600/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1786119683 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5910.516382] Lustre: 201468:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5915.897924] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5915.900587] LDISKFS-fs (dm-0): recovery complete [ 5915.907198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.339038] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5921.299450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:11937) [ 5921.299591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11875 to 0x2c0000401:11905) [ 5927.877336] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5929.114339] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5937.300943] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 12:21:48 (1786119708) [ 5938.257724] LustreError: 202223:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b02f712a680 x1872876205118336/t365072220173(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:367/0 lens 2520/4320 e 0 to 0 dl 1786119722 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5943.594070] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5948.806790] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5949.947699] Lustre: Failing over lustre-MDT0001 [ 5949.956859] LustreError: 211682:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200029441:0x4:0x0].0x0 (ffff8b01c9321500) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5956.223570] Lustre: server umount lustre-MDT0001 complete [ 5973.165227] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5973.168099] LDISKFS-fs (dm-1): recovery complete [ 5973.179399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5975.583808] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5978.624403] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4972 to 0x280000400:4993) [ 5978.624899] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4940 to 0x2c0000400:4961) [ 5978.629163] Lustre: 204256:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b01d1b58000 x1872876242758784/t34359740078(0) o36->ee5672e4-484b-4943-9f37-1d3e916c3f0c@192.168.204.11@tcp:432/0 lens 560/2880 e 0 to 0 dl 1786119787 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5981.440587] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5982.365541] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5986.970018] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 12:22:38 (1786119758) [ 5987.492669] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5987.495897] Lustre: Skipped 1 previous similar message [ 5990.791380] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5994.210089] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5995.067805] Lustre: Failing over lustre-MDT0000 [ 5995.313387] Lustre: server umount lustre-MDT0000 complete [ 6003.679545] Lustre: 212308:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786119760/real 1786119760] req@ffff8b01d20ed500 x1872876205150464/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1786119776 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6010.922096] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6010.923526] LDISKFS-fs (dm-0): recovery complete [ 6010.927662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6012.894638] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6016.584073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:11969) [ 6016.584734] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11875 to 0x2c0000401:11937) [ 6019.161752] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6019.901590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6022.209745] Lustre: Failing over lustre-MDT0001 [ 6022.316225] Lustre: server umount lustre-MDT0001 complete [ 6038.050080] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6038.052348] LDISKFS-fs (dm-1): recovery complete [ 6038.066074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.290384] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6043.656707] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4972 to 0x2c0000400:4993) [ 6043.656715] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5004 to 0x280000400:5025) [ 6046.363338] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6047.229571] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6052.423318] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 12:23:44 (1786119824) [ 6052.959756] LustreError: 201450:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b01c9c2f480 x1872876205207552/t369367187472(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:481/0 lens 2520/4320 e 0 to 0 dl 1786119836 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6052.973222] LustreError: 201450:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 1 previous similar message [ 6059.703564] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6063.692179] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6064.723960] Lustre: Failing over lustre-MDT0000 [ 6064.965808] Lustre: server umount lustre-MDT0000 complete [ 6066.933136] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786119839 with bad export cookie 4349694307265976943 [ 6066.937645] Lustre: Failing over lustre-MDT0001 [ 6066.937758] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6072.799838] LustreError: 217950:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b02d1195f80 x1872876205220608/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6072.813844] LustreError: 217950:0:(client.c:1381:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [ 6072.817757] LustreError: 217950:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6072.960226] Lustre: server umount lustre-MDT0001 complete [ 6089.945416] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6089.946620] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6089.947977] LDISKFS-fs (dm-0): recovery complete [ 6089.955399] LDISKFS-fs (dm-1): recovery complete [ 6089.957400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.976689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6090.158949] LustreError: 219051:0:(llog.c:1655:llog_backup()) MGC192.168.204.111@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6090.164410] Lustre: 219051:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.111@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6091.103173] LustreError: 219053:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 6091.110060] LustreError: 219053:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8b0305b95880 x1872876205221248/t0(0) o250->MGC192.168.204.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786119864 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6091.130803] LustreError: 219053:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 6091.745135] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3c5d37ed4220aa36 [ 6094.699880] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6094.815959] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6097.118290] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5004 to 0x2c0000400:5025) [ 6097.118474] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5036 to 0x280000400:5057) [ 6097.129793] Lustre: 219067:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b02f7188e00 x1872876242830464/t42949673030(0) o36->ee5672e4-484b-4943-9f37-1d3e916c3f0c@192.168.204.11@tcp:551/0 lens 560/2880 e 0 to 0 dl 1786119906 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6097.182950] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11875 to 0x2c0000401:11969) [ 6097.185368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:12001) [ 6100.995466] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6101.966760] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6102.903579] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6108.347728] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 12:24:40 (1786119880) [ 6115.283665] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6116.131870] Lustre: Failing over lustre-MDT0000 [ 6116.392373] Lustre: server umount lustre-MDT0000 complete [ 6125.396424] Lustre: lustre-MDT0001: Client ee5672e4-484b-4943-9f37-1d3e916c3f0c (at 192.168.204.11@tcp) reconnecting [ 6125.401926] Lustre: Skipped 2 previous similar messages [ 6131.972275] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6131.974721] LDISKFS-fs (dm-0): recovery complete [ 6131.979732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.990772] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6137.369894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11875 to 0x2c0000401:12001) [ 6137.372543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:12033) [ 6139.986874] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6140.746277] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6145.458142] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 12:25:17 (1786119917) [ 6145.994489] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6145.996816] Lustre: Skipped 2 previous similar messages [ 6149.423106] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6150.338236] Lustre: Failing over lustre-MDT0001 [ 6150.440541] Lustre: server umount lustre-MDT0001 complete [ 6152.672890] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.672984] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6152.673238] LustreError: 8442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.673246] LustreError: 8442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 150 previous similar messages [ 6152.678092] Lustre: Skipped 45 previous similar messages [ 6152.681613] LustreError: Skipped 9 previous similar messages [ 6166.088823] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6166.091595] LDISKFS-fs (dm-1): recovery complete [ 6166.100576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.103878] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6171.617451] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6171.620226] Lustre: Skipped 49 previous similar messages [ 6171.629080] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6171.635089] Lustre: Skipped 14 previous similar messages [ 6171.658366] Lustre: 219065:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b02fbac0a80 x1872876242896512/t47244640397(0) o36->ee5672e4-484b-4943-9f37-1d3e916c3f0c@192.168.204.11@tcp:600/0 lens 560/2880 e 0 to 0 dl 1786119955 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6171.661780] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5078 to 0x280000400:5121) [ 6171.661812] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5046 to 0x2c0000400:5089) [ 6171.672103] Lustre: 219065:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6174.082113] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6174.762439] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6179.568934] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 12:25:51 (1786119951) [ 6186.900328] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6190.567204] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6191.495606] Lustre: Failing over lustre-MDT0000 [ 6191.696361] Lustre: server umount lustre-MDT0000 complete [ 6195.539447] Lustre: lustre-MDT0001: Client ee5672e4-484b-4943-9f37-1d3e916c3f0c (at 192.168.204.11@tcp) reconnecting [ 6207.603234] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6207.605307] LDISKFS-fs (dm-0): recovery complete [ 6207.610896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6207.768267] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6207.772548] Lustre: Skipped 15 previous similar messages [ 6208.787804] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6208.793100] Lustre: Skipped 14 previous similar messages [ 6209.598771] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6213.143113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:12065) [ 6213.143329] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11875 to 0x2c0000401:12033) [ 6215.705352] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6216.473775] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6218.855522] Lustre: Failing over lustre-MDT0001 [ 6218.972127] Lustre: server umount lustre-MDT0001 complete [ 6234.340602] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6234.343496] LDISKFS-fs (dm-1): recovery complete [ 6234.352903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6234.451804] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (not set up) [ 6234.456067] Lustre: Skipped 11 previous similar messages [ 6236.366192] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6239.744804] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5100 to 0x2c0000400:5121) [ 6239.747503] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5132 to 0x280000400:5153) [ 6242.449117] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6243.264487] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6248.604085] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 12:27:00 (1786120020) [ 6249.096386] LustreError: 219841:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b02fb3d8380 x1872876242967040/t55834574918(0) o36->ee5672e4-484b-4943-9f37-1d3e916c3f0c@192.168.204.11@tcp:678/0 lens 560/448 e 0 to 0 dl 1786120033 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6249.107372] LustreError: 219841:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 3 previous similar messages [ 6255.474591] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6258.979730] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6259.848913] Lustre: Failing over lustre-MDT0000 [ 6259.980831] Lustre: server umount lustre-MDT0000 complete [ 6261.595115] LustreError: 43899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786120034 with bad export cookie 4349694307265990733 [ 6261.596157] Lustre: Failing over lustre-MDT0001 [ 6261.599067] LustreError: 43899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6261.712933] Lustre: server umount lustre-MDT0001 complete [ 6279.059122] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6279.061763] LDISKFS-fs (dm-0): recovery complete [ 6279.075628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.083534] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6279.085823] LDISKFS-fs (dm-1): recovery complete [ 6279.096929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.256613] LustreError: 230210:0:(llog.c:1655:llog_backup()) MGC192.168.204.111@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6279.261665] Lustre: 230210:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.111@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6281.695171] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786120038/real 1786120038] req@ffff8b02f5071880 x1872876205383680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786120054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6281.705175] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6286.815818] LustreError: 3645:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b01c9ca5500 x1872876205386624/t0(0) o250->MGC192.168.204.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6289.198726] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6289.266388] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6293.131350] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5164 to 0x280000400:5185) [ 6293.132266] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5132 to 0x2c0000400:5153) [ 6293.138156] Lustre: 230222:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b02f5845c00 x1872876242967040/t55834574918(0) o36->ee5672e4-484b-4943-9f37-1d3e916c3f0c@192.168.204.11@tcp:722/0 lens 560/2880 e 0 to 0 dl 1786120077 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6293.138165] Lustre: 230222:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6293.182441] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11875 to 0x2c0000401:12065) [ 6293.183103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:12097) [ 6295.975786] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6296.924064] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6297.794637] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6303.070312] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 12:27:54 (1786120074) [ 6307.368485] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6308.229143] Lustre: Failing over lustre-MDT0001 [ 6308.232960] LustreError: 232015:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x240003e99:0xa9:0x0].0x0 (ffff8b01c9321200) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 6314.049428] Lustre: server umount lustre-MDT0001 complete [ 6329.787304] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6329.789170] LDISKFS-fs (dm-1): recovery complete [ 6329.801035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.781702] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6334.986352] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5196 to 0x280000400:5217) [ 6334.986376] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5164 to 0x2c0000400:5185) [ 6338.031682] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6339.012575] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6344.280260] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 12:28:35 (1786120115) [ 6348.893489] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6349.801871] Lustre: Failing over lustre-MDT0000 [ 6350.027347] Lustre: server umount lustre-MDT0000 complete [ 6366.073447] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6366.075664] LDISKFS-fs (dm-0): recovery complete [ 6366.080487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.161197] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6366.164736] LustreError: Skipped 9 previous similar messages [ 6366.295494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.299808] Lustre: Skipped 18 previous similar messages [ 6368.322880] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6371.355995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11875 to 0x2c0000401:12097) [ 6371.356128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:12129) [ 6374.272926] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6375.194731] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6380.746867] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 12:29:12 (1786120152) [ 6384.969863] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6388.038465] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6388.804671] Lustre: Failing over lustre-MDT0000 [ 6389.089895] Lustre: server umount lustre-MDT0000 complete [ 6405.359223] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6405.361302] LDISKFS-fs (dm-0): recovery complete [ 6405.367759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6407.586406] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6410.769618] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11875 to 0x2c0000401:12129) [ 6410.770105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:12161) [ 6413.502941] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6414.190410] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6416.382372] Lustre: Failing over lustre-MDT0001 [ 6416.499891] Lustre: server umount lustre-MDT0001 complete [ 6432.150426] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6432.152446] LDISKFS-fs (dm-1): recovery complete [ 6432.160400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.345189] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6437.383430] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5164 to 0x2c0000400:5217) [ 6437.388308] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5196 to 0x280000400:5249) [ 6440.136896] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6440.849522] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6445.320664] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 12:30:17 (1786120217) [ 6445.872140] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6445.874298] Lustre: Skipped 5 previous similar messages [ 6449.193882] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6452.771413] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6453.675834] Lustre: Failing over lustre-MDT0000 [ 6453.892981] Lustre: server umount lustre-MDT0000 complete [ 6455.678221] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786120228 with bad export cookie 4349694307266001198 [ 6455.682468] Lustre: Failing over lustre-MDT0001 [ 6455.685356] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6455.694520] LustreError: 240262:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200029441:0x1f:0x0].0x0 (ffff8b02f544ff00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6455.703909] LustreError: 240262:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message [ 6460.895879] LustreError: 240262:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b01d2704a80 x1872876205538432/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6460.910240] LustreError: 240262:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6461.039983] Lustre: server umount lustre-MDT0001 complete [ 6476.590403] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6476.591945] LDISKFS-fs (dm-0): recovery complete [ 6476.597380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6476.643114] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6476.646856] LDISKFS-fs (dm-1): recovery complete [ 6476.662278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6480.352187] LustreError: 3645:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b02f535d880 x1872876205539200/t0(0) o250->MGC192.168.204.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6482.470563] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6482.613757] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6486.732637] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11875 to 0x2c0000401:12161) [ 6486.733191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:12193) [ 6486.777082] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5164 to 0x2c0000400:5249) [ 6486.777598] Lustre: 241376:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b02f58fea00 x1872876243046528/t68719476738(0) o36->ee5672e4-484b-4943-9f37-1d3e916c3f0c@192.168.204.11@tcp:186/0 lens 496/2888 e 0 to 0 dl 1786120296 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6486.778137] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5196 to 0x280000400:5281) [ 6486.805929] Lustre: 241376:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6489.119654] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6489.785758] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6490.474860] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6494.471492] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 12:31:06 (1786120266) [ 6498.282840] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6499.058481] Lustre: Failing over lustre-MDT0000 [ 6499.250905] Lustre: server umount lustre-MDT0000 complete [ 6511.442521] Lustre: lustre-MDT0001: Client ee5672e4-484b-4943-9f37-1d3e916c3f0c (at 192.168.204.11@tcp) reconnecting [ 6513.876325] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6513.878449] LDISKFS-fs (dm-0): recovery complete [ 6513.884926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6515.658741] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6519.291626] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11875 to 0x2c0000401:12193) [ 6519.291637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:12225) [ 6521.437505] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6522.017083] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6525.952816] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 12:31:37 (1786120297) [ 6526.400724] LustreError: 242588:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff8b0305bca680 x1872876243073664/t73014444037(0) o36->ee5672e4-484b-4943-9f37-1d3e916c3f0c@192.168.204.11@tcp:200/0 lens 496/456 e 0 to 0 dl 1786120310 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6526.411147] LustreError: 242588:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 5 previous similar messages [ 6529.500903] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6530.244657] Lustre: Failing over lustre-MDT0001 [ 6530.342238] Lustre: server umount lustre-MDT0001 complete [ 6544.723359] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6544.725107] LDISKFS-fs (dm-1): recovery complete [ 6544.730987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6546.394841] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6550.003187] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5164 to 0x2c0000400:5281) [ 6550.003190] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5196 to 0x280000400:5313) [ 6551.987610] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6552.612654] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6556.959969] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 12:32:08 (1786120328) [ 6561.913715] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6569.389865] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6571.518785] Lustre: Failing over lustre-MDT0000 [ 6571.922954] Lustre: server umount lustre-MDT0000 complete [ 6591.967437] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786120348/real 1786120348] req@ffff8b01c7cd8000 x1872876205630080/t0(0) o400->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786120364 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6592.004687] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 6593.299872] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6593.302165] LDISKFS-fs (dm-0): recovery complete [ 6593.307359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6601.186066] LustreError: 3645:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b02f5699180 x1872876205638272/t0(0) o250->MGC192.168.204.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6606.724435] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6607.119288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11875 to 0x2c0000401:12225) [ 6607.123601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:12257) [ 6616.979428] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6618.685776] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6622.490451] Lustre: Failing over lustre-MDT0001 [ 6622.660981] Lustre: server umount lustre-MDT0001 complete [ 6642.005608] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6642.008431] LDISKFS-fs (dm-1): recovery complete [ 6642.017844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6645.866589] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6647.825743] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5164 to 0x2c0000400:5313) [ 6647.826157] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5196 to 0x280000400:5345) [ 6652.498181] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6653.400448] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6659.187927] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 12:33:50 (1786120430) [ 6664.643936] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6669.633385] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6670.810560] Lustre: Failing over lustre-MDT0000 [ 6670.986078] Lustre: server umount lustre-MDT0000 complete [ 6673.519083] LustreError: 151641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786120446 with bad export cookie 4349694307266008807 [ 6673.520493] Lustre: Failing over lustre-MDT0001 [ 6673.536352] LustreError: 151641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6673.704433] Lustre: server umount lustre-MDT0001 complete [ 6693.428327] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6693.430287] LDISKFS-fs (dm-0): recovery complete [ 6693.439433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6693.475468] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6693.479608] LDISKFS-fs (dm-1): recovery complete [ 6693.511493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6698.979765] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3c5d37ed42212072 [ 6703.335979] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6703.391869] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6705.513039] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5164 to 0x2c0000400:5345) [ 6705.516906] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5196 to 0x280000400:5377) [ 6711.208930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11843 to 0x280000401:12289) [ 6711.208958] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11875 to 0x2c0000401:12257) [ 6714.771518] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6715.891598] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6716.877955] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6722.013489] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 12:34:53 (1786120493) [ 6723.068193] Lustre: 253923:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting ee5672e4-484b-4943-9f37-1d3e916c3f0c at adminstrative request [ 6730.136725] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 12:35:01 (1786120501) [ 6734.804485] Lustre: Failing over lustre-MDT0000 [ 6735.028293] Lustre: server umount lustre-MDT0000 complete [ 6753.109458] LustreError: 253352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6753.131438] LustreError: 253352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 225 previous similar messages [ 6753.461375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6760.006658] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6760.007432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12340 to 0x280000401:12385) [ 6760.008192] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12309 to 0x2c0000401:12353) [ 6773.481845] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6777.625930] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6791.398238] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 12:36:01 (1786120561) [ 6814.974914] Lustre: Failing over lustre-OST0000 [ 6814.990963] LustreError: 255588:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x308c:0x0].0x0 (ffff8b02e14f7e00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6815.136302] Lustre: server umount lustre-OST0000 complete [ 6815.222517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6815.247672] Lustre: Skipped 56 previous similar messages [ 6836.418312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6836.598346] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6836.609275] Lustre: Skipped 16 previous similar messages [ 6837.998586] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6838.006876] Lustre: Skipped 16 previous similar messages [ 6838.509097] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6838.514360] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6838.515809] Lustre: Skipped 17 previous similar messages [ 6838.539973] Lustre: Skipped 60 previous similar messages [ 6844.983453] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6855.609763] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6857.516948] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6868.004464] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 12:37:18 (1786120638) [ 6873.267855] Lustre: Failing over lustre-MDT0000 [ 6873.441916] Lustre: server umount lustre-MDT0000 complete [ 6877.666653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6877.673252] LustreError: Skipped 14 previous similar messages [ 6880.497919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6885.524611] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6886.018084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12486 to 0x280000401:12513) [ 6886.019468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12309 to 0x2c0000401:12385) [ 6896.269213] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 12:37:47 (1786120667) [ 6905.732193] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6908.618627] Lustre: Failing over lustre-OST0000 [ 6908.719727] Lustre: server umount lustre-OST0000 complete [ 6932.254752] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6932.257179] LDISKFS-fs (dm-2): recovery complete [ 6932.263880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6939.119427] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6949.200363] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6951.389904] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6963.339635] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 12:38:53 (1786120733) [ 6973.085768] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6977.315325] Lustre: Failing over lustre-OST0000 [ 6977.400938] Lustre: server umount lustre-OST0000 complete [ 7001.801961] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7001.804257] LDISKFS-fs (dm-2): recovery complete [ 7001.812405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7001.958823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7001.963957] Lustre: Skipped 14 previous similar messages [ 7004.188697] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.11@tcp inode [0x2000301a1:0x5:0x0] object 0x280000401:12515 extent [0-4194303]: client csum 14cc0e1c, server csum a53d9726 [ 7009.142238] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7019.586747] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7021.469519] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7030.999613] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 12:40:02 (1786120802) [ 7039.270727] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7048.967601] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7056.638254] Lustre: Failing over lustre-MDT0000 [ 7056.872415] Lustre: server umount lustre-MDT0000 complete [ 7060.720719] LustreError: 151641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786120833 with bad export cookie 4349694307266035869 [ 7060.727817] Lustre: Failing over lustre-OST0000 [ 7060.733911] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7060.742954] LustreError: Skipped 7 previous similar messages [ 7060.962742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7060.968342] Lustre: Skipped 13 previous similar messages [ 7062.833977] Lustre: server umount lustre-OST0000 complete [ 7088.234680] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7088.237557] LDISKFS-fs (dm-0): recovery complete [ 7088.244418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7106.542426] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3c5d37ed42218fbc [ 7112.610860] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7112.927955] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12309 to 0x2c0000401:12417) [ 7139.216594] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7139.228970] LDISKFS-fs (dm-2): recovery complete [ 7139.248685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7147.979474] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7170.794442] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 12:42:21 (1786120941) [ 7183.853140] Lustre: Failing over lustre-OST0000 [ 7184.021735] Lustre: server umount lustre-OST0000 complete [ 7187.959559] Lustre: Failing over lustre-MDT0000 [ 7188.234761] Lustre: server umount lustre-MDT0000 complete [ 7208.798543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.710158] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12309 to 0x2c0000401:12449) [ 7215.680553] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7225.914734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7233.004919] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7235.652349] Lustre: lustre-OST0000: Denying connection for new client 4a6e7b84-d4a3-475d-830f-51ff21072846 (at 192.168.204.11@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:01 [ 7235.681080] Lustre: Skipped 2 previous similar messages [ 7241.047059] Lustre: lustre-OST0000: Denying connection for new client 4a6e7b84-d4a3-475d-830f-51ff21072846 (at 192.168.204.11@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 7246.168430] Lustre: lustre-OST0000: Denying connection for new client 4a6e7b84-d4a3-475d-830f-51ff21072846 (at 192.168.204.11@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:51 [ 7256.408632] Lustre: lustre-OST0000: Denying connection for new client 4a6e7b84-d4a3-475d-830f-51ff21072846 (at 192.168.204.11@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:41 [ 7256.418902] Lustre: Skipped 1 previous similar message [ 7276.886654] Lustre: lustre-OST0000: Denying connection for new client 4a6e7b84-d4a3-475d-830f-51ff21072846 (at 192.168.204.11@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:20 [ 7276.902518] Lustre: Skipped 3 previous similar messages [ 7297.500167] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 7297.505555] Lustre: 267534:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 596f0207-d7df-42f9-8c60-5626729c801f@ [ 7297.511519] Lustre: 267534:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 7297.527251] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 7297.572464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12564 to 0x280000401:12585) [ 7305.731570] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 59 sec [ 7317.850137] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 7326.180803] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 12:44:57 (1786121097) [ 7330.955137] Lustre: Failing over lustre-OST0001 [ 7331.206546] Lustre: server umount lustre-OST0001 complete [ 7352.951183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7360.699735] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7372.938529] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 12:45:43 (1786121143) [ 7376.820825] Lustre: Failing over lustre-OST0000 [ 7376.964994] Lustre: server umount lustre-OST0000 complete [ 7378.806799] LustreError: 43889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7378.853636] LustreError: 43889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 186 previous similar messages [ 7395.949846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7397.550653] LustreError: 270549:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 7397.555369] LustreError: 270549:0:(ldlm_lib.c:2904:target_recovery_thread()) Skipped 77 previous similar messages [ 7403.387985] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7403.487120] Lustre: *** cfs_fail_loc=715, val=40*** [ 7413.727184] Lustre: 3645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786121170/real 1786121170] req@ffff8b02eb54a300 x1872876206092160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786121186 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 7413.752587] Lustre: 3645:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 7413.758090] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [ 7413.766041] Lustre: Skipped 1 previous similar message [ 7419.871592] Lustre: *** cfs_fail_loc=715, val=40*** [ 7419.877989] Lustre: Skipped 2 previous similar messages [ 7420.895373] Lustre: *** cfs_fail_loc=715, val=40*** [ 7420.900438] Lustre: Skipped 1 previous similar message [ 7429.090419] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:09 [ 7429.096719] Lustre: Skipped 2 previous similar messages [ 7430.485130] Lustre: lustre-OST0000: Client 4a6e7b84-d4a3-475d-830f-51ff21072846 (at 192.168.204.11@tcp) reconnected, waiting for 3 clients in recovery for 1:08 [ 7435.231629] Lustre: *** cfs_fail_loc=715, val=40*** [ 7437.575507] LustreError: 270549:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7437.585669] LustreError: 270549:0:(ldlm_lib.c:2904:target_recovery_thread()) Skipped 77 previous similar messages [ 7444.468517] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7446.233427] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7455.153116] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 12:47:06 (1786121226) [ 7459.471364] Lustre: Failing over lustre-MDT0000 [ 7459.740260] Lustre: server umount lustre-MDT0000 complete [ 7459.809966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.848204] Lustre: Skipped 23 previous similar messages [ 7479.400887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7479.742128] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7479.750701] Lustre: Skipped 9 previous similar messages [ 7481.693711] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7481.707177] Lustre: Skipped 9 previous similar messages [ 7484.490535] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7484.909585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7484.927307] Lustre: Skipped 24 previous similar messages [ 7484.951338] LustreError: 272102:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 7487.967254] Lustre: *** cfs_fail_loc=715, val=80*** [ 7487.973609] Lustre: Skipped 2 previous similar messages [ 7498.067274] Lustre: lustre-MDT0000: Client 4a6e7b84-d4a3-475d-830f-51ff21072846 (at 192.168.204.11@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 7500.257326] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7504.352052] Lustre: *** cfs_fail_loc=715, val=80*** [ 7504.357225] Lustre: Skipped 1 previous similar message [ 7514.454656] Lustre: lustre-MDT0000: Client 4a6e7b84-d4a3-475d-830f-51ff21072846 (at 192.168.204.11@tcp) reconnected, waiting for 2 clients in recovery for 0:37 [ 7516.639663] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7520.737770] Lustre: *** cfs_fail_loc=715, val=80*** [ 7520.751705] Lustre: Skipped 1 previous similar message [ 7529.815920] Lustre: lustre-MDT0000: Client 4a6e7b84-d4a3-475d-830f-51ff21072846 (at 192.168.204.11@tcp) reconnected, waiting for 2 clients in recovery for 0:21 [ 7533.025456] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7546.196338] Lustre: lustre-MDT0000: Client 4a6e7b84-d4a3-475d-830f-51ff21072846 (at 192.168.204.11@tcp) reconnected, waiting for 2 clients in recovery for 0:05 [ 7549.413313] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7555.551352] Lustre: *** cfs_fail_loc=715, val=80*** [ 7555.553700] Lustre: Skipped 4 previous similar messages [ 7561.558034] Lustre: lustre-MDT0000: Recovery already passed deadline 0:10. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 7565.023172] LustreError: 272102:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7565.040563] Lustre: 272102:0:(ldlm_lib.c:2950:target_recovery_thread()) too long recovery - read logs [ 7565.048186] LustreError: dumping log to /tmp/lustre-log.1786121338.272102 [ 7565.248383] Lustre: lustre-MDT0000: Recovery over after 1:24, of 2 clients 2 recovered and 0 were evicted. [ 7565.258767] Lustre: Skipped 9 previous similar messages [ 7565.322942] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12462 to 0x2c0000401:12481) [ 7565.323870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12599 to 0x280000401:12617) [ 7574.128954] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7577.435805] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7590.068991] Lustre: DEBUG MARKER: == replay-single test complete, duration 7341 sec ======== 12:49:20 (1786121360) [ 7592.550926] Lustre: DEBUG MARKER: === replay-single: start cleanup 12:49:23 (1786121363) === [ 7605.976732] Lustre: DEBUG MARKER: === replay-single: finish cleanup 12:49:36 (1786121376) === [ 7643.677943] Lustre: server umount lustre-MDT0000 complete [ 7654.321348] LustreError: 151636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786121427 with bad export cookie 4349694307266048637 [ 7654.380294] LustreError: 151636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7654.754089] Lustre: server umount lustre-MDT0001 complete [ 7674.460628] Lustre: server umount lustre-OST0000 complete [ 7684.909779] Lustre: server umount lustre-OST0001 complete [ 7707.201535] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 7710.395538] Key type lgssc unregistered [ 7710.757859] LNet: 275402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7710.767273] LNetError: 275402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7710.784460] LNet: Removed LNI 192.168.204.111@tcp [ 7711.827677] Key type .llcrypt unregistered [ 7711.829983] Key type ._llcrypt unregistered