[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 519890085 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003248] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008722] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011139] LSM: Security Framework initializing [ 0.012052] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.022632] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027292] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028229] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] *** VALIDATE tmpfs *** [ 0.032165] *** VALIDATE proc *** [ 0.033250] *** VALIDATE cgroup *** [ 0.035006] *** VALIDATE cgroup2 *** [ 0.036260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.042819] debug: unmapping init [mem 0xffffffffa4059000-0xffffffffa4060fff] [ 0.044234] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045715] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051019] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053294] rcu: Hierarchical SRCU implementation. [ 0.055466] smp: Bringing up secondary CPUs ... [ 0.056605] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.062515] smp: Brought up 1 node, 4 CPUs [ 0.064017] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.150033] node 0 deferred pages initialised in 83ms [ 0.154110] devtmpfs: initialized [ 0.155249] x86/mm: Memory block size: 128MB [ 0.158111] gcov: version magic: 0x41383552 [ 0.160442] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162259] pinctrl core: initialized pinctrl subsystem [ 0.163188] [ 0.164012] ************************************************************* [ 0.165015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166096] ** ** [ 0.167013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168014] ** ** [ 0.169019] ** This means that this kernel is built to expose internal ** [ 0.170009] ** IOMMU data structures, which may compromise security on ** [ 0.171007] ** your system. ** [ 0.172008] ** ** [ 0.173011] ** If you see this message and you are not debugging the ** [ 0.174013] ** kernel, report this immediately to your vendor! ** [ 0.175012] ** ** [ 0.176008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177010] ************************************************************* [ 0.178643] NET: Registered protocol family 16 [ 0.179358] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.181047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182428] cpuidle: using governor menu [ 0.185149] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188422] PCI: Using configuration type 1 for base access [ 0.190124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.200004] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.205322] cryptd: max_cpu_qlen set to 1000 [ 0.208212] ACPI: Added _OSI(Module Device) [ 0.210018] ACPI: Added _OSI(Processor Device) [ 0.212016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219115] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226801] ACPI: Interpreter enabled [ 0.228060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230014] ACPI: Using IOAPIC for interrupt routing [ 0.232123] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235409] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.246843] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.249103] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.252022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.256111] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.261442] acpiphp: Slot [2] registered [ 0.262127] acpiphp: Slot [5] registered [ 0.264115] acpiphp: Slot [6] registered [ 0.265099] acpiphp: Slot [7] registered [ 0.266094] acpiphp: Slot [8] registered [ 0.268116] acpiphp: Slot [9] registered [ 0.269136] acpiphp: Slot [10] registered [ 0.271152] acpiphp: Slot [3] registered [ 0.272101] acpiphp: Slot [4] registered [ 0.273092] acpiphp: Slot [11] registered [ 0.275184] acpiphp: Slot [12] registered [ 0.276090] acpiphp: Slot [13] registered [ 0.277085] acpiphp: Slot [14] registered [ 0.279107] acpiphp: Slot [15] registered [ 0.281227] acpiphp: Slot [16] registered [ 0.282151] acpiphp: Slot [17] registered [ 0.284091] acpiphp: Slot [18] registered [ 0.285088] acpiphp: Slot [19] registered [ 0.287088] acpiphp: Slot [20] registered [ 0.288129] acpiphp: Slot [21] registered [ 0.289077] acpiphp: Slot [22] registered [ 0.291092] acpiphp: Slot [23] registered [ 0.292140] acpiphp: Slot [24] registered [ 0.294145] acpiphp: Slot [25] registered [ 0.296134] acpiphp: Slot [26] registered [ 0.297142] acpiphp: Slot [27] registered [ 0.299115] acpiphp: Slot [28] registered [ 0.300129] acpiphp: Slot [29] registered [ 0.302144] acpiphp: Slot [30] registered [ 0.304124] acpiphp: Slot [31] registered [ 0.306067] PCI host bridge to bus 0000:00 [ 0.307017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.310024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.316015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.318024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.321023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.323191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326010] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.329317] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.338014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.344044] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.351021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.354545] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356856] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.360038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.363024] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.368014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.380019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.385954] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.391237] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.438018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.450057] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.461015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.473019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.508025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.519872] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.542021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.561020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.605025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.615331] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.626024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.633027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.655030] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.670995] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.677017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.686022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.703024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.716143] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.724024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.734027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.762021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.780427] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.782366] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.784390] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.786440] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.789248] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.794131] iommu: Default domain type: Passthrough [ 0.795466] SCSI subsystem initialized [ 0.797174] ACPI: bus type USB registered [ 0.799110] usbcore: registered new interface driver usbfs [ 0.800128] usbcore: registered new interface driver hub [ 0.802101] usbcore: registered new device driver usb [ 0.804277] pps_core: LinuxPPS API ver. 1 registered [ 0.805013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.808080] PTP clock support registered [ 0.810138] EDAC MC: Ver: 3.0.0 [ 0.811492] PCI: Using ACPI for IRQ routing [ 0.813984] NetLabel: Initializing [ 0.815020] NetLabel: domain hash size = 128 [ 0.816011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.818141] NetLabel: unlabeled traffic allowed by default [ 0.820317] vgaarb: loaded [ 0.821256] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.823012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.831383] clocksource: Switched to clocksource kvm-clock [ 0.943534] VFS: Disk quotas dquot_6.6.0 [ 0.944514] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.945798] *** VALIDATE ramfs *** [ 0.946548] *** VALIDATE hugetlbfs *** [ 0.948126] pnp: PnP ACPI init [ 0.950202] pnp: PnP ACPI: found 6 devices [ 0.971918] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.975442] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.977562] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.979512] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.982140] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.984777] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.987420] NET: Registered protocol family 2 [ 0.989452] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.993920] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.996849] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.001334] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.004128] TCP: Hash tables configured (established 65536 bind 65536) [ 1.006559] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.008811] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.011056] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.013348] NET: Registered protocol family 1 [ 1.015267] RPC: Registered named UNIX socket transport module. [ 1.016859] RPC: Registered udp transport module. [ 1.018516] RPC: Registered tcp transport module. [ 1.020034] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.022556] NET: Registered protocol family 44 [ 1.024281] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.026504] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.028564] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.030925] PCI: CLS 0 bytes, default 64 [ 1.032602] Unpacking initramfs... [ 2.493908] debug: unmapping init [mem 0xffff9f2cbcc54000-0xffff9f2cbffbffff] [ 2.500166] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.502414] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.506469] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.020805] Initialise system trusted keyrings [ 3.022589] Key type blacklist registered [ 3.024464] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.033783] zbud: loaded [ 3.036891] *** VALIDATE nfs *** [ 3.038026] *** VALIDATE nfs4 *** [ 3.039645] pstore: using deflate compression [ 3.043643] Platform Keyring initialized [ 3.148372] NET: Registered protocol family 38 [ 3.150087] Key type asymmetric registered [ 3.151572] Asymmetric key parser 'x509' registered [ 3.153271] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.156216] io scheduler mq-deadline registered [ 3.157702] io scheduler kyber registered [ 3.159207] io scheduler bfq registered [ 3.160820] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.163499] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.166315] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.168824] ACPI: Power Button [PWRF] [ 3.174349] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.180925] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.193976] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.206789] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.221473] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.248541] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.277145] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.282811] Non-volatile memory driver v1.3 [ 3.284520] Linux agpgart interface v0.103 [ 3.321736] virtio_blk virtio1: [vda] 150016 512-byte logical blocks (76.8 MB/73.3 MiB) [ 3.324717] vda: detected capacity change from 0 to 76808192 [ 3.344473] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.347506] vdb: detected capacity change from 0 to 1073741824 [ 3.361532] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.364618] vdc: detected capacity change from 0 to 2621440000 [ 3.379074] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.382126] vdd: detected capacity change from 0 to 2621440000 [ 3.406203] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.408668] vde: detected capacity change from 0 to 4294967296 [ 3.431219] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.433860] vdf: detected capacity change from 0 to 4294967296 [ 3.440360] libphy: Fixed MDIO Bus: probed [ 3.449310] usbcore: registered new interface driver usbserial_generic [ 3.451049] usbserial: USB Serial support registered for generic [ 3.452826] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.458629] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.460988] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.464174] mousedev: PS/2 mouse device common for all mice [ 3.467165] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.467521] rtc_cmos 00:05: RTC can wake from S4 [ 3.474117] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.477813] rtc_cmos 00:05: registered as rtc0 [ 3.480905] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.484992] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.488419] intel_pstate: CPU model not supported [ 3.491890] hid: raw HID events driver (C) Jiri Kosina [ 3.494290] usbcore: registered new interface driver usbhid [ 3.496608] usbhid: USB HID core driver [ 3.498428] drop_monitor: Initializing network drop monitor service [ 3.501271] Initializing XFRM netlink socket [ 3.503489] NET: Registered protocol family 10 [ 3.506989] Segment Routing with IPv6 [ 3.508562] NET: Registered protocol family 17 [ 3.510576] mpls_gso: MPLS GSO support [ 3.516739] RAS: Correctable Errors collector initialized. [ 3.519037] AVX version of gcm_enc/dec engaged. [ 3.520822] AES CTR mode by8 optimization enabled [ 3.614805] sched_clock: Marking stable (3614774619, 0)->(4555378889, -940604270) [ 3.618072] registered taskstats version 1 [ 3.619976] Loading compiled-in X.509 certificates [ 3.622416] zswap: loaded using pool lzo/zbud [ 3.648141] Key type big_key registered [ 3.661065] Key type encrypted registered [ 3.662956] ima: No TPM chip found, activating TPM-bypass! [ 3.665349] ima: Allocated hash algorithm: sha1 [ 3.667317] ima: No architecture policies found [ 3.669302] evm: Initialising EVM extended attributes: [ 3.671295] evm: security.selinux [ 3.672659] evm: security.ima [ 3.673856] evm: security.capability [ 3.675311] evm: HMAC attrs: 0x1 [ 3.677813] rtc_cmos 00:05: setting system clock to 2026-09-12 03:47:11 UTC (1789184831) [ 3.685194] debug: unmapping init [mem 0xffffffffa5003000-0xffffffffa51fffff] [ 3.688342] debug: unmapping init [mem 0xffffffffa3d82000-0xffffffffa4058fff] [ 3.698065] Write protecting the kernel read-only data: 28672k [ 3.701916] debug: unmapping init [mem 0xffffffffa2403000-0xffffffffa25fffff] [ 3.705393] debug: unmapping init [mem 0xffffffffa2d14000-0xffffffffa2dfffff] [ 3.751695] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.760962] systemd[1]: Detected virtualization kvm. [ 3.762937] systemd[1]: Detected architecture x86-64. [ 3.765234] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.795914] systemd[1]: No hostname configured. [ 3.797585] systemd[1]: Set hostname to . [ 3.799752] random: systemd: uninitialized urandom read (16 bytes read) [ 3.802198] systemd[1]: Initializing machine ID from random generator. [ 3.852547] random: ln: uninitialized urandom read (6 bytes read) [ 3.960449] random: systemd: uninitialized urandom read (16 bytes read) [ 3.963554] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.969048] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.974283] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.639790] device-mapper: uevent: version 1.0.3 [ 4.642088] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK [0[ 5.170946] random: fast init done m] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.336897] virtio_net virtio0 ens2: renamed from eth0 [ 5.400691] scsi host0: ata_piix [ 5.405471] scsi host1: ata_piix [ 5.407075] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.409421] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.720586] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.185657] random: crng init done [ 10.187155] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.535369] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.661677] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.895560] SELinux: Disabled at runtime. [ 11.950914] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.960274] systemd[1]: Detected virtualization kvm. [ 11.962278] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.430424] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.432943] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.437447] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.440975] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.444170] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.452490] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.457200] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK [ 12.658470] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.876429] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.122503] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.171607] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.288556] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.296554] EDAC sbridge: Ver: 1.1.2 [ 14.522855] Key type dns_resolver registered [ 14.813635] NFS: Registering the id_resolver key type [ 14.815625] Key type id_resolver registered [ 14.817186] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg142-server login: [ 41.156006] hrtimer: interrupt took 7778185 ns [ 59.129067] libcfs: loading out-of-tree module taints kernel. [ 59.220617] Key type ._llcrypt registered [ 59.223870] Key type .llcrypt registered [ 59.385849] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 78.092170] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 79.838164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 79.865727] alg: No test for adler32 (adler32-zlib) [ 81.356343] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 82.483131] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 84.280689] Key type lgssc registered [ 85.738548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.959391] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 146.264451] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 161.132275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.194483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.501317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 162.600813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 162.839108] Lustre: lustre-MDT0000: new disk, initializing [ 162.984734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.013745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.840347] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.132512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.335119] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 183.408803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 183.418791] Lustre: Skipped 1 previous similar message [ 183.546963] Lustre: lustre-MDT0001: new disk, initializing [ 183.646485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.675363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 183.684902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 188.617625] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.191904] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.903198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.278525] Lustre: lustre-OST0000: new disk, initializing [ 205.284043] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 205.289531] Lustre: 8447:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.354819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.161225] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.538663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 214.545707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 214.624789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.720146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.872812] Lustre: lustre-OST0001: new disk, initializing [ 226.876953] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 226.883600] Lustre: 9521:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.978767] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 234.850578] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.589156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 235.593289] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 235.674189] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 247.739218] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.169479] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 261.171856] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing check_logdir /tmp/testlogs/ [ 267.008521] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing yml_node [ 271.805573] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 274.643626] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 277.659758] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 279.577742] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Fri Sep 11 23:51:45 EDT 2026 [ 299.097711] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 301.465483] Lustre: DEBUG MARKER: === replay-single: start setup 23:52:07 (1789185127) === [ 308.174276] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing check_config_client /mnt/lustre [ 328.569206] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 331.706857] Lustre: 13333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 336.706521] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 342.570299] Lustre: DEBUG MARKER: === replay-single: finish setup 23:52:48 (1789185168) === [ 344.203772] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 23:52:50 (1789185170) [ 352.924681] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 354.859210] Lustre: Failing over lustre-MDT0000 [ 355.155512] Lustre: server umount lustre-MDT0000 complete [ 357.856461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 357.872769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.880064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.889600] Lustre: Skipped 2 previous similar messages [ 360.168887] LustreError: 8705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.196788] LustreError: 8705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 364.011120] LustreError: 6516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.036976] LustreError: 6516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 365.291185] LustreError: 8705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.149462] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.172426] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 374.240348] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.258393] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 375.264176] Lustre: 3641:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185186/real 1789185186] req@ffff9f2c03bcc700 x1876096356306176/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789185202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 375.292280] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 378.842778] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 378.844542] LDISKFS-fs (dm-0): recovery complete [ 378.874414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.506189] LustreError: 7849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 385.509525] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe7df7e322def97ed [ 385.539635] LustreError: 7849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 385.573229] Lustre: MGC192.168.201.142@tcp: Connection restored to 0@lo (at 0@lo) [ 385.893851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 387.064937] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 391.152394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 391.221786] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 391.609769] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 401.854312] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 403.635514] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 412.071745] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 23:53:58 (1789185238) [ 414.827481] Lustre: Failing over lustre-OST0000 [ 414.954602] Lustre: server umount lustre-OST0000 complete [ 416.758091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 416.774429] LustreError: 8431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.779504] Lustre: Skipped 1 previous similar message [ 416.818438] LustreError: 8431:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 435.688849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 435.904365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.473781] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 437.968809] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 437.970726] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 437.978242] Lustre: Skipped 3 previous similar messages [ 443.071060] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 452.638692] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 454.481702] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 464.271431] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 23:54:50 (1789185290) [ 473.058989] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 475.481289] Lustre: Failing over lustre-MDT0000 [ 476.004966] Lustre: server umount lustre-MDT0000 complete [ 477.159955] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.175157] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.192819] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 493.007315] Lustre: 3642:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185304/real 1789185304] req@ffff9f2c027cea00 x1876096356373632/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789185320 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 493.035910] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.323381] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 499.325716] LDISKFS-fs (dm-0): recovery complete [ 499.332140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.247383] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe7df7e322defa369 [ 502.257098] Lustre: MGC192.168.201.142@tcp: Connection restored to 0@lo (at 0@lo) [ 502.268753] Lustre: Skipped 1 previous similar message [ 502.516657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.087446] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 507.873400] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 507.878412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 509.584349] Lustre: lustre-MDT0000: Denying connection for new client 88832c3a-eb7d-4085-9b37-c2fcd07b0efd (at 192.168.201.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 514.803927] Lustre: lustre-MDT0000: Denying connection for new client 88832c3a-eb7d-4085-9b37-c2fcd07b0efd (at 192.168.201.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 519.914236] Lustre: lustre-MDT0000: Denying connection for new client 88832c3a-eb7d-4085-9b37-c2fcd07b0efd (at 192.168.201.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:57 [ 525.052250] Lustre: lustre-MDT0000: Denying connection for new client 88832c3a-eb7d-4085-9b37-c2fcd07b0efd (at 192.168.201.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 530.156503] Lustre: lustre-MDT0000: Denying connection for new client 88832c3a-eb7d-4085-9b37-c2fcd07b0efd (at 192.168.201.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:47 [ 540.394115] Lustre: lustre-MDT0000: Denying connection for new client 88832c3a-eb7d-4085-9b37-c2fcd07b0efd (at 192.168.201.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:37 [ 540.411049] Lustre: Skipped 1 previous similar message [ 560.882251] Lustre: lustre-MDT0000: Denying connection for new client 88832c3a-eb7d-4085-9b37-c2fcd07b0efd (at 192.168.201.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:16 [ 560.909226] Lustre: Skipped 3 previous similar messages [ 577.507857] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 577.524930] Lustre: 18242:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client fa69653e-76ec-4f2c-972d-46390d6ec3f6@ [ 577.539353] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 577.555286] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 577.556710] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 577.575943] Lustre: Skipped 2 previous similar messages [ 577.614503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 577.616480] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 590.758590] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 23:56:56 (1789185416) [ 598.352653] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 600.532237] Lustre: Failing over lustre-MDT0000 [ 600.773955] Lustre: server umount lustre-MDT0000 complete [ 603.108521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 603.115997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.125040] Lustre: Skipped 3 previous similar messages [ 603.127994] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.139501] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 24 previous similar messages [ 621.541728] Lustre: 3642:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185432/real 1789185432] req@ffff9f2c02630380 x1876096356433792/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789185448 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 621.564648] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 623.085935] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 623.088161] LDISKFS-fs (dm-0): recovery complete [ 623.093840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.119306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 632.167338] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 636.587932] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 637.408101] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 637.418594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 638.603235] Lustre: lustre-MDT0000: Denying connection for new client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 638.616292] Lustre: Skipped 3 previous similar messages [ 705.261296] Lustre: lustre-MDT0000: Denying connection for new client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:02 [ 705.293463] Lustre: Skipped 12 previous similar messages [ 707.500651] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 707.508062] Lustre: 19980:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 88832c3a-eb7d-4085-9b37-c2fcd07b0efd@ [ 707.516779] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 707.532497] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 707.534248] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 707.551199] Lustre: Skipped 2 previous similar messages [ 707.577475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 707.577716] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 721.200255] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 23:59:06 (1789185546) [ 730.151897] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 732.702564] Lustre: Failing over lustre-MDT0000 [ 733.154077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 733.167121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.176384] Lustre: Skipped 3 previous similar messages [ 733.183299] LustreError: 7849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 733.202823] LustreError: 7849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 28 previous similar messages [ 733.251385] Lustre: server umount lustre-MDT0000 complete [ 751.059805] Lustre: 3639:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185562/real 1789185562] req@ffff9f2d411da300 x1876096356493952/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789185578 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 751.073177] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 756.529607] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 756.533234] LDISKFS-fs (dm-0): recovery complete [ 756.541195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.299171] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe7df7e322defac14 [ 760.583572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.627198] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 761.579865] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 765.376072] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 765.974244] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 766.022244] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 766.023870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 774.034407] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 775.516345] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 783.784488] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 00:00:10 (1789185610) [ 791.387849] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 793.705730] Lustre: Failing over lustre-MDT0000 [ 793.938834] Lustre: server umount lustre-MDT0000 complete [ 796.639934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 796.643925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.673530] Lustre: Skipped 6 previous similar messages [ 813.023845] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185624/real 1789185624] req@ffff9f2d411c6d80 x1876096356528896/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789185640 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 813.061042] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 816.235745] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 816.238762] LDISKFS-fs (dm-0): recovery complete [ 816.245565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.241396] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2d27b5a680 x1876096356537216/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 822.682822] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 824.048761] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 827.305244] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 827.884395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 827.895328] Lustre: Skipped 5 previous similar messages [ 827.981277] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 828.013611] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 828.014402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 837.245888] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 838.589778] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 847.072306] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 00:01:13 (1789185673) [ 854.346641] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 855.983125] Lustre: Failing over lustre-MDT0000 [ 858.252407] Lustre: server umount lustre-MDT0000 complete [ 858.594880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 858.596200] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.608625] Lustre: Skipped 3 previous similar messages [ 874.976266] Lustre: 3641:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185686/real 1789185686] req@ffff9f2d37283800 x1876096356563712/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789185702 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 875.016822] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 879.988133] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 879.992799] LDISKFS-fs (dm-0): recovery complete [ 880.007386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.518840] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 886.508821] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 888.528547] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 889.935235] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 889.975764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 889.982192] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:193) [ 897.364474] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 898.623383] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 906.400635] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 00:02:12 (1789185732) [ 914.449711] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 916.560560] Lustre: Failing over lustre-MDT0000 [ 916.923553] Lustre: server umount lustre-MDT0000 complete [ 936.928532] Lustre: 3642:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185748/real 1789185748] req@ffff9f2d411d9180 x1876096356598784/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789185764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 936.961304] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 940.057737] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 940.059483] LDISKFS-fs (dm-0): recovery complete [ 940.070503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.170503] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2c03a8f480 x1876096356607360/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 947.438955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 947.444718] Lustre: Skipped 2 previous similar messages [ 947.523781] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 947.947846] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 952.725076] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 952.965801] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 953.001960] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 953.010793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 962.194280] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 963.945701] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 972.134051] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 00:03:18 (1789185798) [ 980.571432] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 982.773805] Lustre: Failing over lustre-MDT0000 [ 983.099623] Lustre: server umount lustre-MDT0000 complete [ 983.520194] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.529351] Lustre: Skipped 6 previous similar messages [ 993.761706] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 993.783325] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 150 previous similar messages [ 999.908183] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185811/real 1789185811] req@ffff9f2d183b2300 x1876096356635264/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789185827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 999.934264] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1004.339159] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1004.342333] LDISKFS-fs (dm-0): recovery complete [ 1004.349092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1010.145671] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2c024e1f80 x1876096356643584/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1010.391202] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1015.794138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1015.807933] Lustre: Skipped 11 previous similar messages [ 1015.873864] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1015.909060] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 1015.915661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1026.690958] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1028.320750] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1037.518347] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 00:04:23 (1789185863) [ 1038.240058] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1038.242794] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1038.246053] LustreError: 6519:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2c03aa4380 x1876096336417408/t38654705666(0) o35->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:467/0 lens 392/456 e 0 to 0 dl 1789185877 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1047.479448] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1049.274297] Lustre: Failing over lustre-MDT0000 [ 1049.681658] Lustre: server umount lustre-MDT0000 complete [ 1051.619874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1067.473312] Lustre: 3641:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185879/real 1789185879] req@ffff9f2d405e4000 x1876096356674560/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789185895 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1067.495399] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1071.596284] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1071.599360] LDISKFS-fs (dm-0): recovery complete [ 1071.609814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.752223] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2d05419500 x1876096356684032/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1079.017963] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.42@tcp (not set up) [ 1079.195819] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1080.631714] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1080.638601] Lustre: Skipped 1 previous similar message [ 1083.035678] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1084.452415] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1084.461326] Lustre: Skipped 1 previous similar message [ 1084.470039] Lustre: 6519:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2d3731c000 x1876096336417408/t38654705666(0) o35->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:513/0 lens 392/456 e 0 to 0 dl 1789185923 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1084.497209] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1084.498502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1091.820325] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1093.313559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1101.658464] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 00:05:27 (1789185927) [ 1108.726690] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1110.713849] Lustre: Failing over lustre-MDT0000 [ 1110.774613] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.42@tcp (stopping) [ 1111.057687] Lustre: server umount lustre-MDT0000 complete [ 1115.105562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1115.112709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.135328] Lustre: Skipped 8 previous similar messages [ 1132.815094] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1132.818534] LDISKFS-fs (dm-0): recovery complete [ 1132.832337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.719826] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe7df7e322defcad0 [ 1141.137689] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1145.662715] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1146.461818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1146.463305] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1154.812671] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1156.431404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1165.891314] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 00:06:32 (1789185992) [ 1173.830223] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1174.666783] Lustre: *** cfs_fail_loc=114, val=0*** [ 1177.417791] Lustre: Failing over lustre-MDT0000 [ 1177.743835] Lustre: server umount lustre-MDT0000 complete [ 1182.189992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1198.562523] Lustre: 3641:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789186010/real 1789186010] req@ffff9f2c03a88380 x1876096356750848/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789186026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1198.610277] Lustre: 3641:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1198.621563] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1198.647795] LustreError: Skipped 1 previous similar message [ 1200.607465] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1200.609332] LDISKFS-fs (dm-0): recovery complete [ 1200.619257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.066153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.071818] Lustre: Skipped 3 previous similar messages [ 1210.106098] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1211.791109] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1211.794377] Lustre: Skipped 1 previous similar message [ 1214.668977] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1215.512512] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1215.521954] Lustre: Skipped 1 previous similar message [ 1215.584276] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1215.584434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1224.430959] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1225.903881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1234.217668] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 00:07:40 (1789186060) [ 1241.015936] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1241.968049] Lustre: *** cfs_fail_loc=128, val=0*** [ 1244.810916] Lustre: Failing over lustre-MDT0000 [ 1245.018322] Lustre: server umount lustre-MDT0000 complete [ 1265.791202] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1265.797283] LDISKFS-fs (dm-0): recovery complete [ 1265.816938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.777521] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2d3f627480 x1876096356796928/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1276.851383] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1277.423688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1277.434258] Lustre: Skipped 16 previous similar messages [ 1277.499472] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1277.515606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1285.752859] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1287.374248] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1295.547995] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 00:08:41 (1789186121) [ 1302.825893] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1305.037203] Lustre: Failing over lustre-MDT0000 [ 1305.366482] Lustre: server umount lustre-MDT0000 complete [ 1308.134727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1308.144739] LustreError: Skipped 1 previous similar message [ 1326.876107] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1326.878890] LDISKFS-fs (dm-0): recovery complete [ 1326.885017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.617536] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1339.733214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1339.735180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1346.439212] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1347.732117] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1355.826761] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 00:09:41 (1789186181) [ 1363.014253] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1364.923136] Lustre: Failing over lustre-MDT0000 [ 1364.973534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1364.985902] Lustre: Skipped 3 previous similar messages [ 1367.141067] Lustre: server umount lustre-MDT0000 complete [ 1387.466777] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1387.471559] LDISKFS-fs (dm-0): recovery complete [ 1387.485102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.924263] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1396.940428] Lustre: Skipped 2 previous similar messages [ 1400.761344] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1402.553471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1402.553557] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1409.254867] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1411.076977] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1419.870823] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 00:10:46 (1789186246) [ 1427.563571] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1435.131505] Lustre: Failing over lustre-MDT0000 [ 1435.400757] Lustre: server umount lustre-MDT0000 complete [ 1438.179544] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.189169] Lustre: Skipped 17 previous similar messages [ 1458.538418] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1458.541715] LDISKFS-fs (dm-0): recovery complete [ 1458.553594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.263907] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2d405e7800 x1876096356910208/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1469.639581] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1473.913046] Lustre: lustre-MDT0000: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 1473.915207] Lustre: Skipped 3 previous similar messages [ 1473.976694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1473.981591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1480.915687] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1482.821567] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1506.904509] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 00:12:13 (1789186333) [ 1514.254592] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1516.305203] Lustre: Failing over lustre-MDT0000 [ 1516.543876] Lustre: server umount lustre-MDT0000 complete [ 1517.307900] LustreError: 6516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.340946] LustreError: 6516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 258 previous similar messages [ 1520.098697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1535.456526] Lustre: 3642:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789186347/real 1789186347] req@ffff9f2d0771ce00 x1876096357002752/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789186363 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1535.494746] Lustre: 3642:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 1535.517348] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1535.534442] LustreError: Skipped 4 previous similar messages [ 1538.663370] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1538.665808] LDISKFS-fs (dm-0): recovery complete [ 1538.672949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1548.036660] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1548.043421] Lustre: Skipped 4 previous similar messages [ 1550.466593] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1551.469887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1551.470406] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1560.296483] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1561.521147] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1572.130785] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 00:13:18 (1789186398) [ 1579.596251] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1581.434253] Lustre: Failing over lustre-MDT0000 [ 1581.657115] Lustre: server umount lustre-MDT0000 complete [ 1603.080358] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1603.082622] LDISKFS-fs (dm-0): recovery complete [ 1603.088869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1607.661440] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe7df7e322df08832 [ 1612.780991] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1613.423645] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1613.430343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1621.624150] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1623.142941] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1630.106275] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 00:14:16 (1789186456) [ 1636.766833] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1638.432346] Lustre: Failing over lustre-MDT0000 [ 1638.679545] Lustre: server umount lustre-MDT0000 complete [ 1660.720376] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1660.724858] LDISKFS-fs (dm-0): recovery complete [ 1660.741444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.048989] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1670.060760] Lustre: Skipped 3 previous similar messages [ 1674.250254] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1675.366227] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1675.366415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1683.007603] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1684.282501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1691.140606] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 00:15:17 (1789186517) [ 1697.317792] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1699.021788] Lustre: Failing over lustre-MDT0000 [ 1699.199583] Lustre: server umount lustre-MDT0000 complete [ 1718.914309] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1718.916192] LDISKFS-fs (dm-0): recovery complete [ 1718.927362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1726.445356] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe7df7e322df092e3 [ 1726.759759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.768325] Lustre: Skipped 7 previous similar messages [ 1731.986914] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1732.256303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1732.258162] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1740.482242] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1741.811503] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1747.948477] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 00:16:14 (1789186574) [ 1753.744984] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1754.979402] Lustre: Failing over lustre-MDT0000 [ 1755.114217] Lustre: server umount lustre-MDT0000 complete [ 1774.547412] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1774.549715] LDISKFS-fs (dm-0): recovery complete [ 1774.557682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.887595] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1789.030498] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1789.032700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1794.632208] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1796.000799] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1803.192559] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 00:17:09 (1789186629) [ 1809.165724] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1810.710308] Lustre: Failing over lustre-MDT0000 [ 1810.908809] Lustre: server umount lustre-MDT0000 complete [ 1814.497785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1814.504151] LustreError: Skipped 2 previous similar messages [ 1830.169406] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1830.171837] LDISKFS-fs (dm-0): recovery complete [ 1830.180992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.576763] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1835.493332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1835.500641] Lustre: Skipped 37 previous similar messages [ 1835.582120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1835.582220] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1842.162320] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1843.476318] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1850.325847] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 00:17:57 (1789186677) [ 1856.529536] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1858.047817] Lustre: Failing over lustre-MDT0000 [ 1858.223319] Lustre: server umount lustre-MDT0000 complete [ 1877.564117] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1877.566517] LDISKFS-fs (dm-0): recovery complete [ 1877.576476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1886.692856] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2d395aad80 x1876096357221376/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1890.762329] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1892.498201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1892.505378] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1899.627751] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1901.083651] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1908.086193] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 00:18:54 (1789186734) [ 1915.426763] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1917.129308] Lustre: Failing over lustre-MDT0000 [ 1917.351516] Lustre: server umount lustre-MDT0000 complete [ 1936.113870] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1936.116452] LDISKFS-fs (dm-0): recovery complete [ 1936.123481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.890663] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1941.578282] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1941.578875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1947.502602] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1948.732236] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1955.728478] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 00:19:42 (1789186782) [ 1962.558176] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1964.372466] Lustre: Failing over lustre-MDT0000 [ 1964.571924] Lustre: server umount lustre-MDT0000 complete [ 1967.078143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.090861] Lustre: Skipped 35 previous similar messages [ 1984.305278] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1984.307329] LDISKFS-fs (dm-0): recovery complete [ 1984.312960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.674275] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2c07ecea00 x1876096357283712/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1996.398766] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1998.358563] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1998.380279] Lustre: Skipped 8 previous similar messages [ 1998.422363] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 1998.424766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 2003.911509] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2005.057204] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2011.672689] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 00:20:38 (1789186838) [ 2017.665612] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2019.333544] Lustre: Failing over lustre-MDT0000 [ 2019.575762] Lustre: server umount lustre-MDT0000 complete [ 2037.888691] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2037.890761] LDISKFS-fs (dm-0): recovery complete [ 2037.899813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2040.747461] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2043.406525] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 2043.409242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:897) [ 2046.816813] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2047.667473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2053.075987] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 00:21:19 (1789186879) [ 2059.038480] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2060.565855] Lustre: Failing over lustre-MDT0000 [ 2060.737793] Lustre: server umount lustre-MDT0000 complete [ 2079.014461] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2079.018062] LDISKFS-fs (dm-0): recovery complete [ 2079.030207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.134819] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2079.143299] LustreError: Skipped 9 previous similar messages [ 2079.206096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2079.214504] Lustre: Skipped 1 previous similar message [ 2081.519448] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2081.523223] Lustre: Skipped 9 previous similar messages [ 2082.369654] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2084.417717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2084.418180] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2089.391116] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2090.543374] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2096.647094] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 00:22:03 (1789186923) [ 2102.013112] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2103.438856] Lustre: Failing over lustre-MDT0000 [ 2103.654184] Lustre: server umount lustre-MDT0000 complete [ 2117.352959] LustreError: 8705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.367681] LustreError: 8705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 305 previous similar messages [ 2120.657319] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789186932/real 1789186932] req@ffff9f2c05aced80 x1876096357363840/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789186948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2120.672283] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 2121.984987] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2121.987190] LDISKFS-fs (dm-0): recovery complete [ 2121.996385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.913461] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2d411c7100 x1876096357372160/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2134.464239] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2136.625541] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 2136.634694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 2141.797354] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2143.162786] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2150.212324] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 00:22:56 (1789186976) [ 2155.491946] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2156.829270] Lustre: Failing over lustre-MDT0000 [ 2157.004222] Lustre: server umount lustre-MDT0000 complete [ 2176.170755] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2176.173534] LDISKFS-fs (dm-0): recovery complete [ 2176.186711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2182.799042] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2182.804794] Lustre: Skipped 9 previous similar messages [ 2186.296401] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2188.317782] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 2188.322577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:993) [ 2193.251803] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2194.451252] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2200.736202] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 00:23:47 (1789187027) [ 2206.783937] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2208.203941] Lustre: Failing over lustre-MDT0000 [ 2208.409779] Lustre: server umount lustre-MDT0000 complete [ 2228.257968] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2228.261694] LDISKFS-fs (dm-0): recovery complete [ 2228.269411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.709951] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2233.902239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2233.902773] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2239.054931] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2240.231509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2247.597987] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 00:24:34 (1789187074) [ 2254.478229] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2256.139248] Lustre: Failing over lustre-MDT0000 [ 2256.450301] Lustre: server umount lustre-MDT0000 complete [ 2276.813897] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2276.818365] LDISKFS-fs (dm-0): recovery complete [ 2276.826255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.047567] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2d0771d500 x1876096357466240/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2286.061512] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 2289.938640] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2291.758925] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2291.761865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2297.439324] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2298.673828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2304.838891] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 00:25:31 (1789187131) [ 2310.285845] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2311.576121] Lustre: Failing over lustre-MDT0000 [ 2311.700949] Lustre: server umount lustre-MDT0000 complete [ 2330.361767] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2330.365543] LDISKFS-fs (dm-0): recovery complete [ 2330.375339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.771979] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe7df7e322df0d627 [ 2337.932670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.938266] Lustre: Skipped 11 previous similar messages [ 2340.614339] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2343.497391] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 2343.497944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 2347.354617] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2348.508763] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2354.428868] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 00:26:21 (1789187181) [ 2357.548927] Lustre: 75339:0:(genops.c:1773:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 at adminstrative request [ 2360.863113] Lustre: Failing over lustre-MDT0000 [ 2361.049345] Lustre: server umount lustre-MDT0000 complete [ 2363.871864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2363.875326] LustreError: Skipped 7 previous similar messages [ 2376.801038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.668447] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2382.360520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2382.363085] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 2386.135611] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2387.157689] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2391.253901] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2397.492215] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2401.697709] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 00:27:08 (1789187228) [ 2402.305096] Lustre: 77361:0:(genops.c:1773:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 at adminstrative request [ 2409.595655] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 00:27:16 (1789187236) [ 2414.396676] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2415.809587] Lustre: Failing over lustre-MDT0000 [ 2415.995888] Lustre: server umount lustre-MDT0000 complete [ 2434.302224] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2434.304903] LDISKFS-fs (dm-0): recovery complete [ 2434.317414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.215379] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2449.386952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2449.391582] Lustre: Skipped 48 previous similar messages [ 2449.515917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1123 to 0x280000401:1153) [ 2449.516064] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1124 to 0x2c0000401:1153) [ 2453.934927] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2454.910825] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2460.757518] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 00:28:07 (1789187287) [ 2466.168263] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2467.546654] Lustre: Failing over lustre-MDT0000 [ 2467.762763] Lustre: server umount lustre-MDT0000 complete [ 2486.625028] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2486.630236] LDISKFS-fs (dm-0): recovery complete [ 2486.652891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2499.471585] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2501.682236] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2501.682337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2506.213526] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2507.262279] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2512.718097] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 00:28:59 (1789187339) [ 2517.232597] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2518.389276] Lustre: Failing over lustre-MDT0000 [ 2518.587835] Lustre: server umount lustre-MDT0000 complete [ 2536.352476] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2536.354215] LDISKFS-fs (dm-0): recovery complete [ 2536.363703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.886636] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2542.128335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2542.134527] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2546.241953] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2547.292300] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2553.038462] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 00:29:39 (1789187379) [ 2557.872841] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2559.090595] Lustre: Failing over lustre-MDT0000 [ 2559.248763] Lustre: server umount lustre-MDT0000 complete [ 2577.604445] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2577.606210] LDISKFS-fs (dm-0): recovery complete [ 2577.613183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.116442] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2583.098737] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2583.112902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2588.247264] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2589.277260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2595.672206] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 00:30:22 (1789187422) [ 2601.148333] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2602.624833] Lustre: Failing over lustre-MDT0000 [ 2602.790778] Lustre: server umount lustre-MDT0000 complete [ 2603.488247] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2603.498187] Lustre: Skipped 51 previous similar messages [ 2621.629393] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2621.630935] LDISKFS-fs (dm-0): recovery complete [ 2621.637340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2632.887944] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2635.771216] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2635.779973] Lustre: Skipped 12 previous similar messages [ 2635.807917] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 2635.811539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 2639.565783] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2640.495865] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2645.957990] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 00:31:12 (1789187472) [ 2650.692828] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2651.995699] Lustre: Failing over lustre-MDT0000 [ 2652.206724] Lustre: server umount lustre-MDT0000 complete [ 2669.231782] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2669.234401] LDISKFS-fs (dm-0): recovery complete [ 2669.241122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.942112] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2674.754729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 2674.755966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 2677.818169] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2678.933715] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2684.903262] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 00:31:51 (1789187511) [ 2689.663727] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2690.922488] Lustre: Failing over lustre-MDT0000 [ 2691.045901] Lustre: server umount lustre-MDT0000 complete [ 2708.677301] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2708.680034] LDISKFS-fs (dm-0): recovery complete [ 2708.686166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.738137] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2708.743212] LustreError: Skipped 12 previous similar messages [ 2711.325662] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2713.320549] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2713.323874] Lustre: Skipped 12 previous similar messages [ 2714.182937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 2714.183729] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 2717.183410] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2717.962698] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2723.101906] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 00:32:30 (1789187550) [ 2727.662858] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2728.709418] Lustre: Failing over lustre-MDT0000 [ 2728.860988] Lustre: server umount lustre-MDT0000 complete [ 2729.444586] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.458169] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 302 previous similar messages [ 2745.823148] Lustre: 3642:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789187557/real 1789187557] req@ffff9f2d370d7800 x1876096357755136/t0(0) o400->MGC192.168.201.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789187573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2745.838270] Lustre: 3642:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 2746.739848] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2746.742714] LDISKFS-fs (dm-0): recovery complete [ 2746.749194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2756.065521] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe7df7e322df108d2 [ 2758.840936] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2761.257704] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 2761.263818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 2764.910807] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2765.853474] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2771.383364] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 00:33:18 (1789187598) [ 2776.172810] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2777.526520] Lustre: Failing over lustre-MDT0000 [ 2777.751889] Lustre: server umount lustre-MDT0000 complete [ 2796.054655] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2796.057154] LDISKFS-fs (dm-0): recovery complete [ 2796.064604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.286653] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2796.292261] Lustre: Skipped 12 previous similar messages [ 2798.720382] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2801.706889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 2801.707058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 2805.510410] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2806.498237] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2812.116735] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 00:33:58 (1789187638) [ 2816.987259] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2818.111735] Lustre: Failing over lustre-MDT0000 [ 2818.287661] Lustre: server umount lustre-MDT0000 complete [ 2834.940134] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2834.942186] LDISKFS-fs (dm-0): recovery complete [ 2834.947262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.391445] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2840.599712] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 2840.600107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 2843.576365] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2844.404144] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2849.349481] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 00:34:36 (1789187676) [ 2854.184563] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2855.180976] Lustre: Failing over lustre-MDT0000 [ 2855.293731] Lustre: server umount lustre-MDT0000 complete [ 2872.814952] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2872.816360] LDISKFS-fs (dm-0): recovery complete [ 2872.822156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.389844] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2878.488898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 2878.489550] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 2881.730311] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2882.633848] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2887.980434] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 00:35:14 (1789187714) [ 2888.643988] Lustre: 98901:0:(genops.c:1773:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 at adminstrative request [ 2895.357436] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 00:35:22 (1789187722) [ 2899.359450] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2900.399688] Lustre: Failing over lustre-MDT0000 [ 2900.530718] Lustre: server umount lustre-MDT0000 complete [ 2907.623591] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2907.624717] LDISKFS-fs (dm-0): recovery complete [ 2907.629309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.852500] Lustre: lustre-MDT0000: Aborting client recovery [ 2907.862308] LustreError: 100181:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2907.866179] LustreError: 100213:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0000-osd: get update log duration 0, retries 0, failed: rc = -108 [ 2907.870176] Lustre: 100215:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2907.878233] Lustre: 100215:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@ [ 2907.883771] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2907.889164] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2907.899749] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2907.936974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1505) [ 2907.940812] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1505) [ 2910.354321] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2913.250633] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2926.012740] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 00:35:52 (1789187752) [ 2930.536645] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2931.730387] Lustre: Failing over lustre-MDT0000 [ 2932.001988] Lustre: server umount lustre-MDT0000 complete [ 2939.802840] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2939.805609] LDISKFS-fs (dm-0): recovery complete [ 2939.818417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.032670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.038131] Lustre: Skipped 13 previous similar messages [ 2940.049826] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2940.071719] Lustre: lustre-MDT0000: Aborting client recovery [ 2940.075294] LustreError: 102044:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2940.082347] Lustre: 102077:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2940.093146] Lustre: 102077:0:(ldlm_lib.c:2438:target_recovery_overseer()) Skipped 2 previous similar messages [ 2940.099280] Lustre: 102077:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2940.108853] Lustre: 102077:0:(genops.c:1600:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2940.114357] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2940.123035] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 2940.134732] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 2940.173473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1512 to 0x280000401:1537) [ 2940.176893] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1512 to 0x2c0000401:1537) [ 2942.835676] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2945.508662] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2953.920505] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2953.922197] Lustre: Skipped 1 previous similar message [ 2958.034719] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 00:36:24 (1789187784) [ 2962.771485] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2963.827366] Lustre: Failing over lustre-MDT0000 [ 2963.969545] Lustre: server umount lustre-MDT0000 complete [ 2965.984201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2965.989868] LustreError: Skipped 13 previous similar messages [ 2971.170878] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2971.172298] LDISKFS-fs (dm-0): recovery complete [ 2971.177164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.383808] Lustre: lustre-MDT0000: Aborting client recovery [ 2971.385593] LustreError: 103909:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2971.385727] Lustre: 103942:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2971.391811] Lustre: 103942:0:(ldlm_lib.c:2438:target_recovery_overseer()) Skipped 2 previous similar messages [ 2971.394143] Lustre: 103942:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2971.398557] Lustre: 103942:0:(genops.c:1600:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2971.402211] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2971.406629] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 2971.414146] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 2971.438962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 2971.443023] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 2973.899306] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2976.737867] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2988.728788] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 00:36:55 (1789187815) [ 2989.309501] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2989.314141] LustreError: 6517:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2d27a92300 x1876096337454336/t201863462916(0) o36->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:153/0 lens 512/456 e 0 to 0 dl 1789187828 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 2992.273318] Lustre: Failing over lustre-MDT0000 [ 2992.397071] Lustre: server umount lustre-MDT0000 complete [ 2996.867974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2997.038808] Lustre: lustre-MDT0000: Aborting client recovery [ 2997.041224] LustreError: 105232:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2997.046574] Lustre: 105264:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2997.055738] Lustre: 105264:0:(ldlm_lib.c:2438:target_recovery_overseer()) Skipped 2 previous similar messages [ 2997.059277] Lustre: 105264:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2997.065158] Lustre: 105264:0:(genops.c:1600:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2997.068474] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2997.077092] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 2997.088827] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 2997.130425] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 2997.131223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1601) [ 2999.459766] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3002.342908] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3012.856741] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3013.677159] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 00:37:20 (1789187840) [ 3017.467884] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3019.092208] Lustre: Failing over lustre-MDT0000 [ 3019.308913] Lustre: server umount lustre-MDT0000 complete [ 3024.954334] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3024.956695] LDISKFS-fs (dm-0): recovery complete [ 3024.962252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3025.124564] Lustre: lustre-MDT0000: Aborting client recovery [ 3025.126465] LustreError: 107192:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3025.132904] Lustre: 107225:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3025.138964] Lustre: 107225:0:(ldlm_lib.c:2438:target_recovery_overseer()) Skipped 2 previous similar messages [ 3025.144979] Lustre: 107225:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3025.149853] Lustre: 107225:0:(genops.c:1600:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3025.153550] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3025.156831] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3025.162855] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3025.164244] Lustre: lustre-MDT0000: Denying connection for new client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 2 evicted) already passed deadline 50:25 [ 3025.189533] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 3025.190176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1633) [ 3027.112504] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3030.497791] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3041.503774] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:37:48 (1789187868) [ 3056.415775] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3057.329433] Lustre: Failing over lustre-MDT0000 [ 3057.463821] Lustre: server umount lustre-MDT0000 complete [ 3072.422232] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3072.424080] LDISKFS-fs (dm-0): recovery complete [ 3072.428088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.428440] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3078.116145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3078.118554] Lustre: Skipped 64 previous similar messages [ 3078.161056] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3078.161681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3080.832263] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3081.616211] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3093.186577] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:38:40 (1789187920) [ 3104.072171] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3107.817412] Lustre: Failing over lustre-MDT0000 [ 3108.020184] Lustre: server umount lustre-MDT0000 complete [ 3123.421328] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3123.422915] LDISKFS-fs (dm-0): recovery complete [ 3123.428299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3125.275365] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3129.819093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3129.819390] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3132.169742] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3132.873635] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3143.493036] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 00:39:30 (1789187970) [ 3144.462246] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3144.889926] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3144.892636] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3147.652408] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 00:39:34 (1789187974) [ 3158.013025] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3162.183388] Lustre: Failing over lustre-OST0000 [ 3162.239872] Lustre: server umount lustre-OST0000 complete [ 3178.434636] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3178.436730] LDISKFS-fs (dm-2): recovery complete [ 3178.442845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3181.826833] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3229.628887] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 00:40:56 (1789188056) [ 3233.280183] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3234.734692] Lustre: Failing over lustre-MDT0000 [ 3234.784726] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3234.785651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3234.795632] Lustre: Skipped 58 previous similar messages [ 3234.797966] Lustre: Skipped 3 previous similar messages [ 3234.881276] Lustre: server umount lustre-MDT0000 complete [ 3250.780601] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3250.781974] LDISKFS-fs (dm-0): recovery complete [ 3250.787911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3252.894341] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3256.314236] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3256.317219] Lustre: Skipped 9 previous similar messages [ 3256.341050] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3256.341146] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3258.634257] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3259.356284] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3272.671250] LustreError: 116543:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3272.674993] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3273.695741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3273.774428] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 00:41:40 (1789188100) [ 3275.984973] LustreError: 6516:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_race id 701 sleeping [ 3281.375127] LustreError: 6516:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3281.378437] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnecting [ 3281.411062] LustreError: 43929:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_fail_race id 701 waking [ 3282.285857] LustreError: 18621:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_race id 701 sleeping [ 3287.519149] LustreError: 18621:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3287.526242] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnecting [ 3288.376576] LustreError: 50564:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_race id 701 sleeping [ 3293.663483] LustreError: 50564:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3293.666900] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnecting [ 3294.550483] LustreError: 6518:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_race id 701 sleeping [ 3299.809109] LustreError: 6518:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3300.673247] LustreError: 8705:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_race id 701 sleeping [ 3305.951105] LustreError: 8705:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3305.955472] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnecting [ 3305.962816] Lustre: Skipped 1 previous similar message [ 3312.997501] LustreError: 50564:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_race id 701 sleeping [ 3313.014153] LustreError: 50564:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 3318.239142] LustreError: 50564:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3318.242624] LustreError: 50564:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 1 previous similar message [ 3324.383141] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnecting [ 3324.385944] Lustre: Skipped 2 previous similar messages [ 3331.316592] LustreError: 18621:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_race id 701 sleeping [ 3331.320940] LustreError: 18621:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 3336.671119] LustreError: 18621:0:(ldlm_lib.c:1185:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3336.675381] LustreError: 18621:0:(ldlm_lib.c:1185:target_handle_connect()) Skipped 2 previous similar messages [ 3340.510788] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 00:42:47 (1789188167) [ 3341.385259] LustreError: 6517:0:(ldlm_lib.c:1443:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3351.786035] Lustre: lustre-MDT0000: Export ffff9f2c04c36000 already connecting from 192.168.201.42@tcp [ 3356.911479] Lustre: lustre-MDT0000: Export ffff9f2c04c36000 already connecting from 192.168.201.42@tcp [ 3362.026310] Lustre: lustre-MDT0000: Export ffff9f2c04c36000 already connecting from 192.168.201.42@tcp [ 3367.145165] Lustre: lustre-MDT0000: Export ffff9f2c04c36000 already connecting from 192.168.201.42@tcp [ 3367.151594] Lustre: Skipped 1 previous similar message [ 3372.265469] Lustre: lustre-MDT0000: Export ffff9f2c04c36000 already connecting from 192.168.201.42@tcp [ 3381.463144] LustreError: 6517:0:(ldlm_lib.c:1443:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3381.467182] Lustre: 6517:0:(service.c:2587:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9f2d06f17100 x1876096340090624/t0(0) o38->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:0/0 lens 520/416 e 0 to 0 dl 1789188189 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3382.508154] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnecting [ 3382.513027] Lustre: Skipped 3 previous similar messages [ 3382.515986] LustreError: 8705:0:(ldlm_lib.c:1443:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3408.109589] Lustre: lustre-MDT0000: Export ffff9f2c04c36000 already connecting from 192.168.201.42@tcp [ 3408.116039] Lustre: Skipped 1 previous similar message [ 3422.567121] LustreError: 8705:0:(ldlm_lib.c:1443:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3422.570662] Lustre: 8705:0:(service.c:2587:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9f2d27c63480 x1876096340095104/t0(0) o38->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:0/0 lens 520/416 e 0 to 0 dl 1789188230 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3423.464504] LustreError: 18621:0:(ldlm_lib.c:1443:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3449.063840] Lustre: lustre-MDT0000: Export ffff9f2c04c36000 already connecting from 192.168.201.42@tcp [ 3449.067019] Lustre: Skipped 2 previous similar messages [ 3463.511130] LustreError: 18621:0:(ldlm_lib.c:1443:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3463.514471] Lustre: 18621:0:(service.c:2587:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9f2d40041500 x1876096340098176/t0(0) o38->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:0/0 lens 520/416 e 0 to 0 dl 1789188271 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3464.426116] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnecting [ 3464.430017] Lustre: Skipped 1 previous similar message [ 3464.431671] LustreError: 6518:0:(ldlm_lib.c:1443:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3490.025210] Lustre: lustre-MDT0000: Export ffff9f2c04c36000 already connecting from 192.168.201.42@tcp [ 3490.029166] Lustre: Skipped 2 previous similar messages [ 3504.479139] LustreError: 6518:0:(ldlm_lib.c:1443:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3504.483198] Lustre: 6518:0:(service.c:2587:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9f2d27a46a00 x1876096340101248/t0(0) o38->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:0/0 lens 520/416 e 0 to 0 dl 1789188312 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3505.389722] LustreError: 6516:0:(ldlm_lib.c:1443:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3545.431142] LustreError: 6516:0:(ldlm_lib.c:1443:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3545.434641] Lustre: 6516:0:(service.c:2587:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9f2d0805b480 x1876096340104448/t0(0) o38->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:0/0 lens 520/416 e 0 to 0 dl 1789188353 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3546.345356] LustreError: 6518:0:(ldlm_lib.c:1443:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3554.465032] LustreError: 6518:0:(ldlm_lib.c:1443:target_handle_connect()) cfs_fail_timeout interrupted [ 3556.217744] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 00:46:23 (1789188383) [ 3559.011540] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3560.411508] Lustre: Failing over lustre-MDT0000 [ 3560.539092] Lustre: server umount lustre-MDT0000 complete [ 3561.713082] LustreError: 18621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3561.719442] LustreError: 18621:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 168 previous similar messages [ 3565.655675] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3565.657340] LDISKFS-fs (dm-0): recovery complete [ 3565.661250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3565.726737] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3565.736402] LustreError: Skipped 12 previous similar messages [ 3565.812366] Lustre: *** cfs_fail_loc=712, val=0*** [ 3565.816547] LustreError: 43934:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9f2d0541aa00 x1876096358747392/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 3565.849623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3565.855984] Lustre: Skipped 7 previous similar messages [ 3565.897419] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3565.897642] Lustre: lustre-MDT0000: Aborting client recovery [ 3565.902366] Lustre: Skipped 21 previous similar messages [ 3565.905597] LustreError: 121388:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3565.910918] Lustre: 121421:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3565.915447] Lustre: 121421:0:(ldlm_lib.c:2438:target_recovery_overseer()) Skipped 2 previous similar messages [ 3565.918632] Lustre: 121421:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3565.922652] Lustre: 121421:0:(genops.c:1600:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3565.925700] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3565.930707] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 3565.936579] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3565.962656] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 3565.964311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 3567.635698] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3571.169283] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3578.393652] Lustre: Failing over lustre-MDT0000 [ 3578.580338] Lustre: server umount lustre-MDT0000 complete [ 3581.411292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3581.415131] LustreError: Skipped 3 previous similar messages [ 3592.038541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.426769] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3592.431673] Lustre: Skipped 8 previous similar messages [ 3594.009267] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3597.319760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 3597.322101] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 3599.676974] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3600.360781] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3604.300127] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 00:47:11 (1789188431) [ 3604.348765] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnecting [ 3604.353464] Lustre: Skipped 2 previous similar messages [ 3607.785562] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 00:47:14 (1789188434) [ 3608.130258] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3608.132130] LustreError: 6526:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2d056f9c00 x1876096340161152/t0(0) o700->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:16/0 lens 264/248 e 0 to 0 dl 1789188446 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 3625.091122] Lustre: Failing over lustre-MDT0000 [ 3625.314375] Lustre: server umount lustre-MDT0000 complete [ 3638.479680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.110996] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3643.894995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 3643.895132] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 3645.944348] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3646.540263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3650.974464] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 00:47:58 (1789188478) [ 3651.956515] Lustre: Failing over lustre-OST0000 [ 3652.041923] Lustre: server umount lustre-OST0000 complete [ 3665.185178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3667.345717] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3671.294553] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3671.868354] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3737.369926] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 00:49:24 (1789188564) [ 3740.203395] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3740.952929] Lustre: Failing over lustre-MDT0000 [ 3741.072636] Lustre: server umount lustre-MDT0000 complete [ 3755.423096] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3755.424832] LDISKFS-fs (dm-0): recovery complete [ 3755.431847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.990565] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3760.609363] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3760.611018] Lustre: Skipped 29 previous similar messages [ 3760.753496] Lustre: *** cfs_fail_loc=216, val=0*** [ 3760.753906] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 3760.755407] LustreError: 127539:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 3761.826288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 3823.386502] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 00:50:50 (1789188650) [ 3831.463767] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 00:50:58 (1789188658) [ 3832.392593] Lustre: Failing over lustre-MDT0000 [ 3832.547828] Lustre: server umount lustre-MDT0000 complete [ 3837.407961] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.412258] Lustre: Skipped 25 previous similar messages [ 3845.234889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3846.671300] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3850.729142] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 3850.730520] LustreError: 129121:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2d27a45c00 x1876096340280064/t0(0) o101->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:301/0 lens 328/344 e 0 to 0 dl 1789188731 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 3908.338722] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnected, waiting for 2 clients in recovery for 1:05 [ 3908.346665] Lustre: 129121:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3908.354694] Lustre: lustre-MDT0000: Recovery over after 1:00, of 2 clients 2 recovered and 0 were evicted. [ 3908.357782] Lustre: Skipped 4 previous similar messages [ 3908.372134] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 3908.372133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 3910.353868] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3910.878328] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3914.468272] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 00:52:21 (1789188741) [ 3915.854266] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3918.595145] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3919.192798] Lustre: Failing over lustre-MDT0000 [ 3919.378814] Lustre: server umount lustre-MDT0000 complete [ 3933.515325] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3933.516855] LDISKFS-fs (dm-0): recovery complete [ 3933.520217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3935.116968] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3938.809871] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 3938.809876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3137) [ 3940.663891] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3941.235573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3944.850713] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 00:52:52 (1789188772) [ 3945.234924] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3948.853847] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3949.476886] Lustre: Failing over lustre-MDT0000 [ 3949.561983] Lustre: server umount lustre-MDT0000 complete [ 3963.644547] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3963.646366] LDISKFS-fs (dm-0): recovery complete [ 3963.650749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3965.226496] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3969.013109] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3137) [ 3969.013356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 3970.776170] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3971.287583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3974.735839] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 00:53:21 (1789188801) [ 3975.134567] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3978.720831] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3979.346141] Lustre: Failing over lustre-MDT0000 [ 3979.435499] Lustre: server umount lustre-MDT0000 complete [ 3993.514072] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3993.515325] LDISKFS-fs (dm-0): recovery complete [ 3993.518040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3994.907845] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3998.706019] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 3998.706027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3201) [ 4003.046660] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 00:53:50 (1789188830) [ 4004.384376] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4004.385769] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4004.386983] LustreError: 6520:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2d082ae300 x1876096340331648/t257698037777(0) o35->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:413/0 lens 392/456 e 0 to 0 dl 1789188843 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4005.288800] Lustre: Failing over lustre-MDT0000 [ 4005.526537] Lustre: server umount lustre-MDT0000 complete [ 4018.184241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.539347] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4023.790193] Lustre: 39732:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2d3958f100 x1876096340331648/t257698037777(0) o35->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:432/0 lens 392/456 e 0 to 0 dl 1789188862 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4023.794664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 4023.794678] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3201) [ 4025.548970] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4026.078301] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4029.339626] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 00:54:16 (1789188856) [ 4029.683349] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4029.684671] LustreError: 6516:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2d40735c00 x1876096340344448/t261993005072(0) o36->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:482/0 lens 504/448 e 0 to 0 dl 1789188912 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4033.300653] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4033.911116] Lustre: Failing over lustre-MDT0000 [ 4033.991523] Lustre: server umount lustre-MDT0000 complete [ 4047.903829] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4047.905194] LDISKFS-fs (dm-0): recovery complete [ 4047.908454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4049.519218] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4053.486453] Lustre: 8705:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2c03a89f80 x1876096340344448/t261993005072(0) o36->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:506/0 lens 504/2880 e 0 to 0 dl 1789188936 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4053.492981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 4053.493325] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3233) [ 4055.291826] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4055.803494] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4059.126680] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 00:54:46 (1789188886) [ 4059.491638] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4059.493277] LustreError: 6518:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2c09dcd880 x1876096340358144/t266287972368(0) o36->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:512/0 lens 504/448 e 0 to 0 dl 1789188942 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4060.787900] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4063.148491] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4063.769639] Lustre: Failing over lustre-MDT0000 [ 4063.854252] Lustre: server umount lustre-MDT0000 complete [ 4077.958432] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4077.960213] LDISKFS-fs (dm-0): recovery complete [ 4077.963451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4079.444470] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4083.183273] Lustre: 50564:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2d27b0e300 x1876096340358144/t266287972368(0) o36->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:535/0 lens 504/2880 e 0 to 0 dl 1789188965 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4083.197219] Lustre: 50564:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4083.197736] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 4083.197741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3297) [ 4087.623555] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 00:55:14 (1789188914) [ 4087.969937] LustreError: 8705:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2d411c6a00 x1876096340370432/t270582939664(0) o36->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:540/0 lens 504/448 e 0 to 0 dl 1789188970 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4087.978684] LustreError: 8705:0:(ldlm_lib.c:3382:target_send_reply_msg()) Skipped 1 previous similar message [ 4089.259366] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4089.261196] Lustre: Skipped 1 previous similar message [ 4091.993329] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4092.658129] Lustre: Failing over lustre-MDT0000 [ 4092.734235] Lustre: server umount lustre-MDT0000 complete [ 4106.862291] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4106.863591] LDISKFS-fs (dm-0): recovery complete [ 4106.866710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.458111] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4112.367378] Lustre: 6516:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2d3958c000 x1876096340370432/t270582939664(0) o36->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:565/0 lens 504/2880 e 0 to 0 dl 1789188995 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4112.372685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3329) [ 4112.372686] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 4116.296332] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 00:55:43 (1789188943) [ 4116.629630] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4117.911492] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4117.913029] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4117.914741] Lustre: Skipped 2 previous similar messages [ 4117.916545] LustreError: 6519:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2d27b15500 x1876096340382720/t274877906960(0) o35->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:526/0 lens 392/456 e 0 to 0 dl 1789188956 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4121.315941] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4121.899810] Lustre: Failing over lustre-MDT0000 [ 4121.982374] Lustre: server umount lustre-MDT0000 complete [ 4135.782590] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4135.783750] LDISKFS-fs (dm-0): recovery complete [ 4135.786039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.210291] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4141.038560] Lustre: 39732:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2d3f625f80 x1876096340382720/t274877906960(0) o35->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:549/0 lens 392/456 e 0 to 0 dl 1789188979 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4141.044255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3361) [ 4141.044260] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329) [ 4145.251249] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 00:56:12 (1789188972) [ 4145.564719] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4205.295772] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnecting [ 4205.299868] Lustre: Skipped 4 previous similar messages [ 4205.304026] Lustre: 6517:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2d0c3c8380 x1876096340392576/t279172874255(0) o101->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:658/0 lens 664/3488 e 0 to 0 dl 1789189088 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4207.800358] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 00:57:15 (1789189035) [ 4210.483145] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4211.140322] Lustre: Failing over lustre-MDT0000 [ 4211.220654] Lustre: server umount lustre-MDT0000 complete [ 4212.703921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4212.704076] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4212.706318] LustreError: Skipped 9 previous similar messages [ 4212.711417] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 176 previous similar messages [ 4225.614735] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4225.616413] LDISKFS-fs (dm-0): recovery complete [ 4225.620764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.654949] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4225.658560] LustreError: Skipped 12 previous similar messages [ 4225.727181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4225.730386] Lustre: Skipped 13 previous similar messages [ 4225.760143] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4225.763478] Lustre: Skipped 15 previous similar messages [ 4225.768232] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4225.771924] Lustre: Skipped 12 previous similar messages [ 4227.183623] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4231.160483] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 4231.160484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3393) [ 4232.999541] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4233.541851] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4247.003328] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 00:57:54 (1789189074) [ 4249.838692] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4250.480130] Lustre: Failing over lustre-MDT0000 [ 4250.689941] Lustre: server umount lustre-MDT0000 complete [ 4264.784337] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4264.785948] LDISKFS-fs (dm-0): recovery complete [ 4264.790296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4266.371105] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4270.074990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 4270.075040] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3393) [ 4271.986075] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4272.558804] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4274.923558] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4280.127426] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 00:58:27 (1789189107) [ 4290.111043] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4290.698759] Lustre: Failing over lustre-MDT0000 [ 4291.024763] Lustre: server umount lustre-MDT0000 complete [ 4304.963490] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4304.964577] LDISKFS-fs (dm-0): recovery complete [ 4304.967029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4306.416395] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4310.580303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 4310.580309] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 4312.377956] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4312.892558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4326.309177] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 00:59:13 (1789189153) [ 4330.132840] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4330.752178] Lustre: Failing over lustre-MDT0000 [ 4330.900365] Lustre: server umount lustre-MDT0000 complete [ 4344.818864] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4344.821011] LDISKFS-fs (dm-0): recovery complete [ 4344.823996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4346.163017] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4350.458445] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 4350.458456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 4352.189243] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4352.705032] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4356.085482] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 4356.583990] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 4358.711802] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 00:59:45 (1789189185) [ 4364.555647] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 4425.214072] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4425.215265] Lustre: Skipped 1 previous similar message [ 4425.216371] LustreError: 8705:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2d06f8e050 x1876096343181824/t296352743434(0) o36->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:123/0 lens 66040/440 e 0 to 0 dl 1789189308 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 4425.225673] LustreError: 8705:0:(ldlm_lib.c:3382:target_send_reply_msg()) Skipped 1 previous similar message [ 4484.842149] Lustre: 6518:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2d4046d500 x1876096343181824/t296352743434(0) o36->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:182/0 lens 66040/440 e 0 to 0 dl 1789189367 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 4487.673933] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 4488.233267] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 01:01:55 (1789189315) [ 4493.059224] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4494.099083] Lustre: Failing over lustre-MDT0000 [ 4494.410943] Lustre: server umount lustre-MDT0000 complete [ 4498.911501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.916263] Lustre: Skipped 49 previous similar messages [ 4508.748555] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4508.750163] LDISKFS-fs (dm-0): recovery complete [ 4508.753942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4510.342091] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4514.273225] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4514.275517] Lustre: Skipped 57 previous similar messages [ 4514.491673] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4514.495075] Lustre: Skipped 12 previous similar messages [ 4514.513635] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 4514.513650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 4516.441806] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4516.953000] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4520.561906] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 01:02:27 (1789189347) [ 4526.181537] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4529.163417] Lustre: Failing over lustre-OST0000 [ 4529.202273] Lustre: server umount lustre-OST0000 complete [ 4543.219989] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 4543.221442] LDISKFS-fs (dm-2): recovery complete [ 4543.224424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4545.243139] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4556.771445] Lustre: Failing over lustre-OST0000 [ 4556.820747] Lustre: server umount lustre-OST0000 complete [ 4569.577698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4571.561798] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4575.252649] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4575.774851] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4609.842887] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 01:03:57 (1789189437) [ 4610.798152] Lustre: Failing over lustre-MDT0000 [ 4611.049507] Lustre: server umount lustre-MDT0000 complete [ 4623.905914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4625.351249] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4629.487254] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 4629.487259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 4636.872767] Lustre: Failing over lustre-MDT0000 [ 4637.001152] Lustre: server umount lustre-MDT0000 complete [ 4650.008514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.538806] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4655.603473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 4655.603664] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 4657.382463] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4657.912439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4661.480232] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 01:04:48 (1789189488) [ 4672.919222] Lustre: Failing over lustre-OST0000 [ 4672.971338] Lustre: server umount lustre-OST0000 complete [ 4685.766369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4687.726656] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4691.322904] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4691.807583] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4695.371791] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 01:05:22 (1789189522) [ 4695.985954] Lustre: Failing over lustre-MDT0000 [ 4696.032317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4696.243889] Lustre: server umount lustre-MDT0000 complete [ 4699.209804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4699.244409] Lustre: *** cfs_fail_loc=605, val=0*** [ 4699.245543] LustreError: 164473:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc088f340 failed: rc = -95 [ 4699.248595] LustreError: 164473:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 4699.250373] LustreError: 164473:0:(obd_mount.c:259:lustre_start_simple()) MGS setup error -95 [ 4699.252421] LustreError: 164473:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 4699.254148] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 4699.256287] LustreError: 164473:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 4699.261149] Lustre: server umount lustre-MDT0000 complete [ 4699.262278] LustreError: 164473:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 4701.803154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4703.221523] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4705.799669] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 01:05:33 (1789189533) [ 4707.316640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 4707.316686] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 4709.731673] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4710.661863] Lustre: Failing over lustre-MDT0000 [ 4710.997445] Lustre: server umount lustre-MDT0000 complete [ 4724.896506] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4724.897701] LDISKFS-fs (dm-0): recovery complete [ 4724.902682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.480859] Lustre: *** cfs_fail_loc=707, val=0*** [ 4726.265353] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4785.914982] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 4785.925887] Lustre: 166642:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4785.928874] Lustre: 166642:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 1 previous similar message [ 4786.019478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 4786.019478] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 4787.911397] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4788.387184] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4792.037267] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 01:06:59 (1789189619) [ 4815.777285] LustreError: 6517:0:(service.c:2563:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f2d3f5f6300 x1876096344060800/t0(0) o101->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:469/0 lens 664/0 e 0 to 0 dl 1789189654 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4815.791968] LustreError: 6517:0:(service.c:2564:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 4826.823107] LustreError: 6517:0:(service.c:2564:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4826.828575] LustreError: 39732:0:(service.c:2563:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f2d40268700 x1876096344062080/t0(0) o35->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:480/0 lens 392/0 e 0 to 0 dl 1789189665 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4832.225208] LustreError: 41871:0:(service.c:2563:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f2d40472680 x1876096360385408/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:486/0 lens 544/0 e 0 to 0 dl 1789189671 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 4832.232580] LustreError: 41871:0:(service.c:2563:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 4838.649553] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 01:07:45 (1789189665) [ 4862.364088] LustreError: 8436:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 4873.391143] LustreError: 8436:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 4876.381961] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 01:08:23 (1789189703) [ 4899.358495] LustreError: 18621:0:(service.c:2563:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f2d40476a00 x1876096344084864/t0(0) o101->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:553/0 lens 576/0 e 0 to 0 dl 1789189738 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4899.367271] LustreError: 18621:0:(service.c:2564:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 4904.471171] LustreError: 18621:0:(service.c:2564:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4904.476943] LustreError: 50564:0:(service.c:2563:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f2d0805aa00 x1876096344085504/t0(0) o101->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:558/0 lens 664/0 e 0 to 0 dl 1789189743 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4915.455067] LustreError: 6516:0:(service.c:2564:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4915.462485] LustreError: 6517:0:(service.c:2563:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f2d0832c700 x1876096344107392/t0(0) o101->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:608/0 lens 664/0 e 0 to 0 dl 1789189793 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4915.471497] LustreError: 6517:0:(service.c:2563:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 4927.415187] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 01:09:14 (1789189754) [ 5022.554414] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 01:10:49 (1789189849) [ 5045.510705] LustreError: 6517:0:(service.c:2563:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f2d055ac000 x1876096344163200/t0(0) o101->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:699/0 lens 576/0 e 0 to 0 dl 1789189884 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5045.520651] LustreError: 6517:0:(service.c:2563:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 5045.523529] LustreError: 6517:0:(service.c:2564:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5045.526250] LustreError: 6517:0:(service.c:2564:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 5045.943088] LustreError: 6517:0:(service.c:2564:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5061.776796] LustreError: 6518:0:(service.c:2564:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5061.779600] LustreError: 6518:0:(service.c:2564:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 5062.199047] LustreError: 6518:0:(service.c:2564:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5062.200903] LustreError: 6518:0:(service.c:2564:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 5077.628387] LustreError: 6516:0:(service.c:2563:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f2d40737b80 x1876096344182400/t0(0) o36->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:731/0 lens 504/0 e 0 to 0 dl 1789189916 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 5077.641356] LustreError: 6516:0:(service.c:2563:ptlrpc_server_handle_request()) Skipped 79 previous similar messages [ 5089.953769] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 01:11:57 (1789189917) [ 5114.512929] Lustre: DEBUG MARKER: phase 2 [ 5117.690110] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 01:12:24 (1789189944) [ 5187.846984] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 01:13:35 (1789190015) [ 5188.382130] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 5188.972544] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 01:13:36 (1789190016) [ 5190.780298] Lustre: DEBUG MARKER: Started rundbench load pid=135058 ... [ 5194.702714] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5196.219033] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 5196.906537] Lustre: Failing over lustre-MDT0000 [ 5197.161824] Lustre: server umount lustre-MDT0000 complete [ 5198.759943] LustreError: 6516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5198.764611] LustreError: 6516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 136 previous similar messages [ 5200.863376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5200.866440] LustreError: Skipped 9 previous similar messages [ 5200.868659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.874810] Lustre: Skipped 24 previous similar messages [ 5211.693813] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5211.695711] LDISKFS-fs (dm-0): recovery complete [ 5211.700370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5211.744462] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5211.748028] LustreError: Skipped 8 previous similar messages [ 5211.828603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.832425] Lustre: Skipped 11 previous similar messages [ 5211.846442] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5211.849678] Lustre: Skipped 11 previous similar messages [ 5211.879488] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5211.882903] Lustre: Skipped 11 previous similar messages [ 5213.270601] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5217.251375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5217.254454] Lustre: Skipped 25 previous similar messages [ 5217.261288] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 5217.265222] Lustre: Skipped 7 previous similar messages [ 5217.280867] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 5217.280952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 5219.297431] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5219.889022] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5224.784255] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5226.346833] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 5227.000588] Lustre: Failing over lustre-MDT0001 [ 5227.004978] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.42@tcp (stopping) [ 5227.007313] Lustre: Skipped 3 previous similar messages [ 5227.189326] Lustre: server umount lustre-MDT0001 complete [ 5241.354390] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 5241.355487] LDISKFS-fs (dm-1): recovery complete [ 5241.359219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5242.892665] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5247.261051] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:744 to 0x2c0000400:769) [ 5247.261068] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:761 to 0x280000400:801) [ 5249.087383] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5249.654802] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5254.498417] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5256.027340] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 5256.693943] Lustre: Failing over lustre-MDT0000 [ 5256.937029] Lustre: server umount lustre-MDT0000 complete [ 5271.829218] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5271.830899] LDISKFS-fs (dm-0): recovery complete [ 5271.835659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.567910] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5277.187417] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 5277.187680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 5279.129981] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5279.696786] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5284.760350] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5286.325834] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 5287.025529] Lustre: Failing over lustre-MDT0001 [ 5287.206212] Lustre: server umount lustre-MDT0001 complete [ 5301.245813] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5301.247540] LDISKFS-fs (dm-1): recovery complete [ 5301.251199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5302.659410] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5307.266938] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1124 to 0x280000400:1153) [ 5307.266959] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1091 to 0x2c0000400:1121) [ 5309.111314] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5309.633873] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5336.443596] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 01:16:03 (1789190163) [ 5459.242206] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5470.019653] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 5470.616811] Lustre: Failing over lustre-MDT0000 [ 5470.688744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5470.691867] Lustre: Skipped 2 previous similar messages [ 5470.857467] Lustre: server umount lustre-MDT0000 complete [ 5484.772547] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5484.774315] LDISKFS-fs (dm-0): recovery complete [ 5484.777747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5486.321886] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5496.394688] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6549 to 0x2c0000401:6593) [ 5496.397959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6581 to 0x280000401:6625) [ 5499.861041] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5500.900084] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5626.117671] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5637.033381] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 5637.932613] Lustre: Failing over lustre-MDT0000 [ 5638.007835] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.42@tcp (stopping) [ 5638.011083] Lustre: Skipped 1 previous similar message [ 5643.744345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5643.747771] Lustre: Skipped 8 previous similar messages [ 5643.864517] Lustre: server umount lustre-MDT0000 complete [ 5659.722797] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5659.724663] LDISKFS-fs (dm-0): recovery complete [ 5659.728989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5661.568860] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5668.175269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7372 to 0x280000401:7393) [ 5668.175405] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7340 to 0x2c0000401:7361) [ 5670.685506] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5671.391473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5707.914721] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 01:22:15 (1789190535) [ 5830.903498] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5841.834113] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 5842.534178] Lustre: Failing over lustre-MDT0001 [ 5842.557122] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.42@tcp (stopping) [ 5842.776654] Lustre: server umount lustre-MDT0001 complete [ 5843.937668] LustreError: 9520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190671 with bad export cookie 16708211896555587340 [ 5843.942613] LustreError: 9520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5843.944099] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5843.949983] Lustre: Skipped 21 previous similar messages [ 5843.951765] LustreError: 8440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.956846] LustreError: 8440:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 79 previous similar messages [ 5857.414199] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5857.415651] LDISKFS-fs (dm-1): recovery complete [ 5857.420136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.546352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5857.549290] Lustre: Skipped 5 previous similar messages [ 5857.557880] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5857.563112] Lustre: Skipped 5 previous similar messages [ 5858.918165] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5859.370711] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5859.373651] Lustre: Skipped 5 previous similar messages [ 5862.881740] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5862.884242] Lustre: Skipped 21 previous similar messages [ 5863.785849] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5863.789432] Lustre: Skipped 5 previous similar messages [ 5863.804440] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4040 to 0x280000400:4065) [ 5863.804485] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4008 to 0x2c0000400:4033) [ 5865.614642] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5866.188222] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5990.297341] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6000.992394] Lustre: DEBUG MARKER: test_70d fail mds2 2 times [ 6001.616975] Lustre: Failing over lustre-MDT0001 [ 6001.862687] Lustre: server umount lustre-MDT0001 complete [ 6002.144335] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6002.146436] LustreError: Skipped 3 previous similar messages [ 6016.002233] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6016.003858] LDISKFS-fs (dm-1): recovery complete [ 6016.007909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.446625] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6022.759171] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4460 to 0x280000400:4481) [ 6022.759291] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4429 to 0x2c0000400:4449) [ 6024.617754] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6025.222407] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6028.587059] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 01:27:35 (1789190855) [ 6151.552604] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6162.327021] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 6162.993238] Lustre: Failing over lustre-MDT0001 [ 6163.190580] Lustre: server umount lustre-MDT0001 complete [ 6177.274723] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6177.275987] LDISKFS-fs (dm-1): recovery complete [ 6177.279216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.385565] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6178.683027] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6182.901476] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4492 to 0x280000400:4513) [ 6182.901490] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4460 to 0x2c0000400:4481) [ 6184.636382] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6185.157684] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6308.989867] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6319.716803] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 6320.305781] Lustre: Failing over lustre-MDT0000 [ 6320.335841] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.42@tcp (stopping) [ 6320.523654] Lustre: server umount lustre-MDT0000 complete [ 6334.522141] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6334.523374] LDISKFS-fs (dm-0): recovery complete [ 6334.526978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.577837] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6334.580769] LustreError: Skipped 3 previous similar messages [ 6336.020116] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6342.697802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12728 to 0x280000401:12769) [ 6342.701967] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12695 to 0x2c0000401:12737) [ 6344.597168] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6345.157800] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6348.958920] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 01:32:56 (1789191176) [ 6354.840690] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6356.352575] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 6356.996837] Lustre: Failing over lustre-OST0000 [ 6357.044646] Lustre: server umount lustre-OST0000 complete [ 6370.800953] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6370.802152] LDISKFS-fs (dm-2): recovery complete [ 6370.804776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6372.726715] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6376.560872] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6377.100144] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6385.868721] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6387.342944] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 6387.946234] Lustre: Failing over lustre-OST0000 [ 6387.987748] Lustre: server umount lustre-OST0000 complete [ 6401.702200] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6401.703277] LDISKFS-fs (dm-2): recovery complete [ 6401.705373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6403.524784] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6407.389318] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6407.946883] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6413.308156] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 01:34:00 (1789191240) [ 6536.065558] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6538.516838] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6549.251186] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 6549.883608] Lustre: Failing over lustre-MDT0000 [ 6549.927334] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.42@tcp (stopping) [ 6550.089459] Lustre: server umount lustre-MDT0000 complete [ 6550.495566] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6550.496028] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6550.499332] Lustre: Skipped 19 previous similar messages [ 6550.502934] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 71 previous similar messages [ 6551.283557] LustreError: 151844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191379 with bad export cookie 16708211896559773487 [ 6551.284554] Lustre: Failing over lustre-MDT0001 [ 6551.287815] LustreError: 151844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6551.485578] Lustre: server umount lustre-MDT0001 complete [ 6566.101287] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6566.103468] LDISKFS-fs (dm-1): recovery complete [ 6566.108543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6566.119163] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6566.122415] LDISKFS-fs (dm-0): recovery complete [ 6566.126927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6566.187355] LustreError: 198289:0:(llog.c:1655:llog_backup()) MGC192.168.201.142@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6566.190873] Lustre: 198289:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.142@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6571.999215] Lustre: 3639:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789191383/real 1789191383] req@ffff9f2c05c86d80 x1876096395841024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789191399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6572.006927] Lustre: 3639:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6577.119354] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2c0ca4b480 x1876096395843968/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6577.191143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6577.192846] Lustre: Skipped 4 previous similar messages [ 6577.198892] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6577.200446] Lustre: Skipped 5 previous similar messages [ 6578.347513] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6578.350945] Lustre: Skipped 6 previous similar messages [ 6578.682789] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6578.739748] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6582.752669] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6582.754862] Lustre: Skipped 16 previous similar messages [ 6583.335222] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6583.339879] Lustre: Skipped 5 previous similar messages [ 6583.356450] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4515 to 0x280000400:4545) [ 6583.356648] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4483 to 0x2c0000400:4513) [ 6588.771916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:12865) [ 6588.771927] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12811 to 0x2c0000401:12833) [ 6590.639838] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6591.203576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6591.762957] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6715.639346] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6718.117176] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6728.815113] Lustre: DEBUG MARKER: fail mds1 mds2 2 times [ 6729.460873] Lustre: Failing over lustre-MDT0000 [ 6729.461149] Lustre: 198297:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff9f2c0dd79500 x1876096406481536/t0(0) o104->lustre-MDT0000@192.168.201.42@tcp:15/16 lens 328/224 e 0 to 1 dl 0 ref 1 fl Rpc:EeXQU/0/ffffffff rc -19/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 [ 6729.471556] Lustre: 198297:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 6729.523076] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.42@tcp (stopping) [ 6729.644440] Lustre: server umount lustre-MDT0000 complete [ 6730.826568] LustreError: 9520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191558 with bad export cookie 16708211896560780080 [ 6730.827779] Lustre: Failing over lustre-MDT0001 [ 6730.830654] LustreError: 9520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6731.068839] Lustre: server umount lustre-MDT0001 complete [ 6745.820591] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6745.823881] LDISKFS-fs (dm-0): recovery complete [ 6745.828593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6745.847336] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6745.850027] LDISKFS-fs (dm-1): recovery complete [ 6745.855048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6755.807343] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2c0c68b480 x1876096406486272/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6755.986963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 6755.991685] LustreError: Skipped 3 previous similar messages [ 6757.340552] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6757.474064] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6759.088899] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4483 to 0x2c0000400:4545) [ 6759.088902] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4515 to 0x280000400:4577) [ 6772.839151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:12897) [ 6772.839160] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12811 to 0x2c0000401:12865) [ 6774.884193] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6775.416877] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6776.007873] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6779.466469] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 01:40:06 (1789191606) [ 6781.869210] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6782.761793] Lustre: Failing over lustre-MDT0000 [ 6782.905994] Lustre: server umount lustre-MDT0000 complete [ 6796.824371] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6796.825814] LDISKFS-fs (dm-0): recovery complete [ 6796.828686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6798.057231] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 6798.235430] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6813.424437] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 6813.435948] Lustre: 201675:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2c028a4a80 x1876096427326592/t339302417128(339302417128) o101->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:202/0 lens 520/3488 e 0 to 0 dl 1789191652 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6813.471399] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:12897) [ 6813.471399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:12929) [ 6815.423886] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6815.922086] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6819.391400] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 01:40:46 (1789191646) [ 6821.856547] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6822.803065] Lustre: Failing over lustre-MDT0000 [ 6823.021904] Lustre: server umount lustre-MDT0000 complete [ 6837.100232] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6837.101370] LDISKFS-fs (dm-0): recovery complete [ 6837.104035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6838.513362] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6839.026234] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 6839.027780] LustreError: 203605:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2c0c87d180 x1876096427326592/t339302417128(339302417128) o101->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:227/0 lens 520/664 e 0 to 0 dl 1789191677 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6854.386868] Lustre: lustre-MDT0000: Client 798b652a-7ab3-49d5-80ad-dfaad7ef3fe9 (at 192.168.201.42@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 6854.391137] Lustre: 201676:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2c0d768e00 x1876096427326592/t339302417128(339302417128) o101->798b652a-7ab3-49d5-80ad-dfaad7ef3fe9@192.168.201.42@tcp:243/0 lens 520/3488 e 0 to 0 dl 1789191693 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6854.417433] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12899 to 0x2c0000401:12929) [ 6854.417442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:12961) [ 6856.412394] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6856.918853] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6860.376947] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 01:41:27 (1789191687) [ 6861.927643] Lustre: Failing over lustre-OST0000 [ 6861.969407] Lustre: server umount lustre-OST0000 complete [ 6863.048459] Lustre: Failing over lustre-MDT0000 [ 6863.139321] Lustre: server umount lustre-MDT0000 complete [ 6875.865458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.138439] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6877.820972] Lustre: lustre-MDT0000: Denying connection for new client 7fcee186-3f75-4441-a115-29c4485897ca (at 192.168.201.42@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 6881.264220] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12899 to 0x2c0000401:12961) [ 6884.984124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6886.186679] Lustre: lustre-OST0000: Denying connection for new client 7fcee186-3f75-4441-a115-29c4485897ca (at 192.168.201.42@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 6887.056129] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6890.470722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:12993) [ 6891.229494] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 01:41:58 (1789191718) [ 6891.578357] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6891.580677] LustreError: 202460:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2c0e40df80 x1876096408305792/t360777252871(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:280/0 lens 1312/4320 e 0 to 0 dl 1789191730 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6894.093444] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6894.721024] Lustre: Failing over lustre-MDT0000 [ 6895.038439] Lustre: server umount lustre-MDT0000 complete [ 6907.871139] Lustre: 201728:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789191719/real 1789191719] req@ffff9f2c0e40ed80 x1876096408305792/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1789191735 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6907.879256] Lustre: 201728:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 16 previous similar messages [ 6908.933351] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6908.934464] LDISKFS-fs (dm-0): recovery complete [ 6908.937572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.314904] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6914.547748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:13025) [ 6914.547755] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12963 to 0x2c0000401:12993) [ 6916.169688] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6916.658232] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6919.981916] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 01:42:27 (1789191747) [ 6920.295159] LustreError: 208072:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2c0e40ea00 x1876096408334848/t365072220172(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:309/0 lens 2520/4320 e 0 to 0 dl 1789191759 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6922.574742] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6924.798844] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6925.338863] Lustre: Failing over lustre-MDT0001 [ 6925.342652] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.42@tcp (stopping) [ 6925.344729] Lustre: Skipped 4 previous similar messages [ 6931.015930] Lustre: server umount lustre-MDT0001 complete [ 6945.202666] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6945.204458] LDISKFS-fs (dm-1): recovery complete [ 6945.209086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6946.661685] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6950.386897] Lustre: 203605:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2d3aa1dc00 x1876096441580160/t34359741195(0) o36->7fcee186-3f75-4441-a115-29c4485897ca@192.168.201.42@tcp:361/0 lens 560/2880 e 0 to 0 dl 1789191811 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6950.390448] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4556 to 0x2c0000400:4577) [ 6950.395249] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4588 to 0x280000400:4609) [ 6952.155713] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6952.691946] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6956.432630] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 01:43:03 (1789191783) [ 6956.783665] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6956.785526] Lustre: Skipped 1 previous similar message [ 6956.786708] LustreError: 202460:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2c04a7c000 x1876096408367744/t365072220193(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:345/0 lens 2520/4320 e 0 to 0 dl 1789191795 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6959.247740] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6961.797753] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6962.422850] Lustre: Failing over lustre-MDT0000 [ 6962.677982] Lustre: server umount lustre-MDT0000 complete [ 6976.809258] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6976.810581] LDISKFS-fs (dm-0): recovery complete [ 6976.813480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.846884] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6976.849602] LustreError: Skipped 6 previous similar messages [ 6978.297623] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6982.157932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:13057) [ 6982.158180] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12963 to 0x2c0000401:13025) [ 6984.027977] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6984.574504] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6986.545267] Lustre: Failing over lustre-MDT0001 [ 6986.633963] Lustre: server umount lustre-MDT0001 complete [ 7000.830336] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7000.831659] LDISKFS-fs (dm-1): recovery complete [ 7000.836398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7002.287714] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7006.194791] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4588 to 0x2c0000400:4609) [ 7006.194791] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4620 to 0x280000400:4641) [ 7007.931475] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7008.479544] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7012.168904] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 01:43:59 (1789191839) [ 7018.028832] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7020.545979] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7021.199308] Lustre: Failing over lustre-MDT0000 [ 7021.442214] Lustre: server umount lustre-MDT0000 complete [ 7022.720915] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191850 with bad export cookie 16708211896561769012 [ 7022.722185] Lustre: Failing over lustre-MDT0001 [ 7022.723946] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7029.215827] LustreError: 218189:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9f2c0d768700 x1876096408433152/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7029.222511] LustreError: 218189:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7029.354547] Lustre: server umount lustre-MDT0001 complete [ 7044.237453] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7044.240023] LDISKFS-fs (dm-0): recovery complete [ 7044.243106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.250245] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7044.252479] LDISKFS-fs (dm-1): recovery complete [ 7044.257393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7048.159479] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2c04a7ea00 x1876096408433920/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7049.763895] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7049.826325] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7053.448665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:13089) [ 7053.448742] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12963 to 0x2c0000401:13057) [ 7053.462158] Lustre: 220074:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2c04a7fb80 x1876096441648000/t42949673028(0) o36->7fcee186-3f75-4441-a115-29c4485897ca@192.168.201.42@tcp:467/0 lens 560/2880 e 0 to 0 dl 1789191917 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7053.463239] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4652 to 0x280000400:4673) [ 7053.463303] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4620 to 0x2c0000400:4641) [ 7055.280535] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7055.829412] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7056.333842] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7059.817641] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 01:44:47 (1789191887) [ 7060.166415] LustreError: 220076:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2c09fa1180 x1876096441680256/t47244640324(0) o36->7fcee186-3f75-4441-a115-29c4485897ca@192.168.201.42@tcp:487/0 lens 560/448 e 0 to 0 dl 1789191937 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7060.173907] LustreError: 220076:0:(ldlm_lib.c:3382:target_send_reply_msg()) Skipped 1 previous similar message [ 7065.579634] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7066.197304] Lustre: Failing over lustre-MDT0000 [ 7066.445324] Lustre: server umount lustre-MDT0000 complete [ 7080.481987] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7080.483169] LDISKFS-fs (dm-0): recovery complete [ 7080.485820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7082.003635] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7086.073836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:13121) [ 7086.073875] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12963 to 0x2c0000401:13089) [ 7087.838453] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7088.415552] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7114.486143] Lustre: lustre-MDT0001: Client 7fcee186-3f75-4441-a115-29c4485897ca (at 192.168.201.42@tcp) reconnecting [ 7114.489119] Lustre: Skipped 2 previous similar messages [ 7114.492120] Lustre: 219305:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2c0d425180 x1876096441680256/t47244640324(0) o36->7fcee186-3f75-4441-a115-29c4485897ca@192.168.201.42@tcp:542/0 lens 560/2880 e 0 to 0 dl 1789191992 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7116.977554] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 01:45:44 (1789191944) [ 7117.346335] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7117.347698] Lustre: Skipped 2 previous similar messages [ 7119.794254] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7120.411688] Lustre: Failing over lustre-MDT0001 [ 7120.506130] Lustre: server umount lustre-MDT0001 complete [ 7134.598565] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7134.600077] LDISKFS-fs (dm-1): recovery complete [ 7134.604790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7136.219408] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7139.825491] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4694 to 0x280000400:4737) [ 7139.825565] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4662 to 0x2c0000400:4705) [ 7141.704502] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7142.245572] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7145.979790] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 01:46:13 (1789191973) [ 7151.800282] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7154.350287] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7154.964854] Lustre: Failing over lustre-MDT0000 [ 7155.167950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7155.171451] Lustre: Skipped 46 previous similar messages [ 7155.289061] Lustre: server umount lustre-MDT0000 complete [ 7155.438072] LustreError: 219305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.442231] LustreError: 219305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 150 previous similar messages [ 7169.456432] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7169.457783] LDISKFS-fs (dm-0): recovery complete [ 7169.460538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7170.910947] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7174.658059] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12963 to 0x2c0000401:13121) [ 7174.658059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:13153) [ 7176.517248] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7177.047731] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7179.017890] Lustre: Failing over lustre-MDT0001 [ 7179.114869] Lustre: server umount lustre-MDT0001 complete [ 7193.310613] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7193.311960] LDISKFS-fs (dm-1): recovery complete [ 7193.316304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7193.415585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7193.418349] Lustre: Skipped 16 previous similar messages [ 7193.426933] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7193.430181] Lustre: Skipped 16 previous similar messages [ 7194.727093] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7194.795378] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7194.798189] Lustre: Skipped 15 previous similar messages [ 7198.689911] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7198.692097] Lustre: Skipped 55 previous similar messages [ 7198.696620] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7198.700288] Lustre: Skipped 16 previous similar messages [ 7198.717438] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4716 to 0x2c0000400:4737) [ 7198.717438] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4748 to 0x280000400:4769) [ 7200.508269] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7201.003963] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7204.542672] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 01:47:11 (1789192031) [ 7204.888223] LustreError: 219304:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2d3a277100 x1876096441779072/t55834574916(0) o36->7fcee186-3f75-4441-a115-29c4485897ca@192.168.201.42@tcp:593/0 lens 560/448 e 0 to 0 dl 1789192043 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7204.895687] LustreError: 219304:0:(ldlm_lib.c:3382:target_send_reply_msg()) Skipped 2 previous similar messages [ 7210.398217] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7212.890706] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7213.531939] Lustre: Failing over lustre-MDT0000 [ 7213.824088] Lustre: server umount lustre-MDT0000 complete [ 7215.060710] LustreError: 151844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789192042 with bad export cookie 16708211896561782725 [ 7215.061747] Lustre: Failing over lustre-MDT0001 [ 7215.066447] LustreError: 151844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7215.174893] Lustre: server umount lustre-MDT0001 complete [ 7230.267262] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7230.270225] LDISKFS-fs (dm-0): recovery complete [ 7230.272644] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7230.274436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7230.274650] LDISKFS-fs (dm-1): recovery complete [ 7230.282251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7234.527150] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789192046/real 1789192046] req@ffff9f2d3a275c00 x1876096408602240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789192062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7234.538932] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7239.647402] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2c0e40aa00 x1876096408604160/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7241.409513] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7241.545704] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7245.958540] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12963 to 0x2c0000401:13153) [ 7245.958552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:13185) [ 7246.965814] Lustre: 230473:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2c09fa0a80 x1876096441779072/t55834574916(0) o36->7fcee186-3f75-4441-a115-29c4485897ca@192.168.201.42@tcp:635/0 lens 560/2880 e 0 to 0 dl 1789192085 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7246.971671] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4801) [ 7246.971680] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4748 to 0x2c0000400:4769) [ 7246.974612] Lustre: 230473:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 2 previous similar messages [ 7248.792680] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7249.307926] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7249.825302] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7253.356300] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 01:48:00 (1789192080) [ 7256.204909] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7256.864855] Lustre: Failing over lustre-MDT0001 [ 7256.868912] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.42@tcp (stopping) [ 7256.871377] Lustre: Skipped 14 previous similar messages [ 7263.286871] Lustre: server umount lustre-MDT0001 complete [ 7277.415720] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7277.418053] LDISKFS-fs (dm-1): recovery complete [ 7277.422894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7278.935142] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7282.679261] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4801) [ 7282.679277] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4812 to 0x280000400:4833) [ 7284.509351] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7285.062027] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7288.592596] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 01:48:35 (1789192115) [ 7291.454495] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7292.054935] Lustre: Failing over lustre-MDT0000 [ 7292.313734] Lustre: server umount lustre-MDT0000 complete [ 7306.274548] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7306.275872] LDISKFS-fs (dm-0): recovery complete [ 7306.278841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7307.682725] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7311.869863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:13217) [ 7311.869882] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12963 to 0x2c0000401:13185) [ 7313.633353] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7314.210107] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7317.681562] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 01:49:04 (1789192144) [ 7320.532438] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7323.114751] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7323.778742] Lustre: Failing over lustre-MDT0000 [ 7324.119771] Lustre: server umount lustre-MDT0000 complete [ 7338.054450] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7338.055689] LDISKFS-fs (dm-0): recovery complete [ 7338.060246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7339.454021] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7343.612159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:13249) [ 7343.612190] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12963 to 0x2c0000401:13217) [ 7345.322477] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7345.878173] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7347.815741] Lustre: Failing over lustre-MDT0001 [ 7347.897809] Lustre: server umount lustre-MDT0001 complete [ 7361.943717] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7361.945056] LDISKFS-fs (dm-1): recovery complete [ 7361.949194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7363.288083] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7367.155827] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4833) [ 7367.155889] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4812 to 0x280000400:4865) [ 7368.900652] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7369.401249] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7372.789146] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 01:50:00 (1789192200) [ 7375.624473] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7378.122287] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7378.731663] Lustre: Failing over lustre-MDT0000 [ 7379.105362] Lustre: server umount lustre-MDT0000 complete [ 7380.285474] LustreError: 151842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789192208 with bad export cookie 16708211896561793141 [ 7380.286640] Lustre: Failing over lustre-MDT0001 [ 7380.288577] LustreError: 151842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7380.290737] LustreError: 240509:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200029441:0x1f:0x0].0x0 (ffff9f2d05682000) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7386.079906] LustreError: 240509:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9f2d3190b800 x1876096408747776/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7386.085287] LustreError: 240509:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7386.194729] Lustre: server umount lustre-MDT0001 complete [ 7401.160135] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7401.163947] LDISKFS-fs (dm-1): recovery complete [ 7401.170422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7401.171136] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7401.175414] LDISKFS-fs (dm-0): recovery complete [ 7401.179112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7401.252059] LustreError: 241603:0:(llog.c:1655:llog_backup()) MGC192.168.201.142@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7401.256146] Lustre: 241603:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.142@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7406.186774] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 7406.189834] LustreError: Skipped 16 previous similar messages [ 7407.581793] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7407.664709] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7411.251670] Lustre: 241616:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f2c0bf18000 x1876096441855744/t68719476738(0) o36->7fcee186-3f75-4441-a115-29c4485897ca@192.168.201.42@tcp:71/0 lens 496/2888 e 0 to 0 dl 1789192276 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 7411.257924] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4812 to 0x280000400:4897) [ 7411.258380] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4865) [ 7411.260359] Lustre: 241616:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7411.296437] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12963 to 0x2c0000401:13249) [ 7411.296447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:13281) [ 7413.032552] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7413.523433] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7414.020482] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7417.477839] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 01:50:44 (1789192244) [ 7417.855957] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7417.857570] Lustre: Skipped 6 previous similar messages [ 7420.576330] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7421.189188] Lustre: Failing over lustre-MDT0000 [ 7421.463946] Lustre: server umount lustre-MDT0000 complete [ 7433.448059] Lustre: lustre-MDT0001: Client 7fcee186-3f75-4441-a115-29c4485897ca (at 192.168.201.42@tcp) reconnecting [ 7433.451235] Lustre: Skipped 1 previous similar message [ 7435.443515] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7435.444886] LDISKFS-fs (dm-0): recovery complete [ 7435.447645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7436.876990] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7440.891808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:13313) [ 7440.891823] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12963 to 0x2c0000401:13281) [ 7442.618969] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7443.146138] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7446.633065] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 01:51:13 (1789192273) [ 7449.428217] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7450.035257] Lustre: Failing over lustre-MDT0001 [ 7450.118587] Lustre: server umount lustre-MDT0001 complete [ 7463.929366] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7463.930579] LDISKFS-fs (dm-1): recovery complete [ 7463.933716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.311179] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7469.559432] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4812 to 0x280000400:4929) [ 7469.559486] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4897) [ 7471.314801] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7471.859396] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7475.147143] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 01:51:42 (1789192302) [ 7475.484562] LustreError: 242396:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9f2d395e9180 x1876096441892608/t77309411330(0) o36->7fcee186-3f75-4441-a115-29c4485897ca@192.168.201.42@tcp:109/0 lens 496/456 e 0 to 0 dl 1789192314 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 7475.491688] LustreError: 242396:0:(ldlm_lib.c:3382:target_send_reply_msg()) Skipped 6 previous similar messages [ 7477.835727] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7480.226178] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7480.825124] Lustre: Failing over lustre-MDT0000 [ 7480.981931] Lustre: server umount lustre-MDT0000 complete [ 7495.004144] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7495.005402] LDISKFS-fs (dm-0): recovery complete [ 7495.008031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7496.494069] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7500.294381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:13345) [ 7500.294404] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12963 to 0x2c0000401:13313) [ 7502.235189] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7502.778985] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7504.764194] Lustre: Failing over lustre-MDT0001 [ 7504.847905] Lustre: server umount lustre-MDT0001 complete [ 7518.935256] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7518.936674] LDISKFS-fs (dm-1): recovery complete [ 7518.940248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.433398] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7524.339394] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4929) [ 7524.339419] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4812 to 0x280000400:4961) [ 7526.128156] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7526.658714] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7530.105336] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 01:52:37 (1789192357) [ 7532.868683] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7535.285443] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7535.920139] Lustre: Failing over lustre-MDT0000 [ 7536.179476] Lustre: server umount lustre-MDT0000 complete [ 7537.330605] LustreError: 151844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789192365 with bad export cookie 16708211896561800694 [ 7537.331958] Lustre: Failing over lustre-MDT0001 [ 7537.334473] LustreError: 151844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7537.427263] Lustre: server umount lustre-MDT0001 complete [ 7552.434977] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7552.438505] LDISKFS-fs (dm-0): recovery complete [ 7552.439478] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7552.444046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7552.444989] LDISKFS-fs (dm-1): recovery complete [ 7552.454798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7562.207380] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2d31908a80 x1876096408885376/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7562.214070] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 7563.985412] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7564.163603] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7571.709623] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4961) [ 7571.709674] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4812 to 0x280000400:4993) [ 7576.839986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12843 to 0x280000401:13377) [ 7576.839986] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12963 to 0x2c0000401:13345) [ 7578.815579] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7579.368870] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7579.897470] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7583.316259] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 01:53:30 (1789192410) [ 7583.930716] Lustre: 254185:0:(genops.c:1773:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 7fcee186-3f75-4441-a115-29c4485897ca at adminstrative request [ 7588.697225] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 01:53:35 (1789192415) [ 7590.585193] Lustre: Failing over lustre-MDT0000 [ 7590.747218] Lustre: server umount lustre-MDT0000 complete [ 7603.500401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7603.540676] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7603.543599] LustreError: Skipped 10 previous similar messages [ 7604.940304] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7608.846443] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13397 to 0x2c0000401:13441) [ 7608.846506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13428 to 0x280000401:13473) [ 7610.608807] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7611.096704] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7614.540061] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 01:54:01 (1789192441) [ 7618.454125] Lustre: Failing over lustre-OST0000 [ 7618.521080] Lustre: server umount lustre-OST0000 complete [ 7631.290355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7633.281481] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7637.024796] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7637.590833] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7641.140139] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 01:54:28 (1789192468) [ 7642.368979] Lustre: Failing over lustre-MDT0000 [ 7642.621465] Lustre: server umount lustre-MDT0000 complete [ 7645.062970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7646.475777] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7647.171132] Lustre: lustre-MDT0000: Denying connection for new client b207b558-d47a-47d1-bebc-de5890f3ef50 (at 192.168.201.42@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 7647.177198] Lustre: Skipped 1 previous similar message [ 7650.293972] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13397 to 0x2c0000401:13473) [ 7650.294075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13428 to 0x280000401:13505) [ 7654.687425] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 01:54:41 (1789192481) [ 7657.419742] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7658.279743] Lustre: Failing over lustre-OST0000 [ 7658.313508] Lustre: server umount lustre-OST0000 complete [ 7672.743837] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7672.745137] LDISKFS-fs (dm-2): recovery complete [ 7672.747906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7674.749193] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7678.368171] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7678.928960] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7682.385180] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 01:55:09 (1789192509) [ 7685.109223] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7686.944227] Lustre: Failing over lustre-OST0000 [ 7686.980750] Lustre: server umount lustre-OST0000 complete [ 7700.799239] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7700.800685] LDISKFS-fs (dm-2): recovery complete [ 7700.804388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7702.705926] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.201.42@tcp inode [0x2000301a1:0x5:0x0] object 0x280000401:13507 extent [0-4194303]: client csum ca361b09, server csum 105c78fc [ 7702.779124] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7706.418342] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7706.939707] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7710.507177] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 01:55:37 (1789192537) [ 7712.987099] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7715.423142] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7717.311341] Lustre: Failing over lustre-MDT0000 [ 7717.651286] Lustre: server umount lustre-MDT0000 complete [ 7718.829514] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789192546 with bad export cookie 16708211896561826811 [ 7718.831825] Lustre: Failing over lustre-OST0000 [ 7718.834864] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7718.877558] Lustre: server umount lustre-OST0000 complete [ 7733.199683] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7733.201451] LDISKFS-fs (dm-0): recovery complete [ 7733.204135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7743.968558] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe7df7e322e860dd6 [ 7745.329223] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7749.200749] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13397 to 0x2c0000401:13505) [ 7759.328178] LustreError: 43934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7759.333613] LustreError: 43934:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 221 previous similar messages [ 7759.530284] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7759.531551] LDISKFS-fs (dm-2): recovery complete [ 7759.534127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7761.531465] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7768.589224] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 01:56:35 (1789192595) [ 7773.322294] Lustre: Failing over lustre-OST0000 [ 7773.371665] Lustre: server umount lustre-OST0000 complete [ 7774.621771] Lustre: Failing over lustre-MDT0000 [ 7774.687574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7774.691962] Lustre: Skipped 66 previous similar messages [ 7774.693893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7774.695628] Lustre: Skipped 14 previous similar messages [ 7774.998368] Lustre: server umount lustre-MDT0000 complete [ 7787.821042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7789.352719] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7793.148276] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13397 to 0x2c0000401:13537) [ 7798.490570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7798.549054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7798.551480] Lustre: Skipped 22 previous similar messages [ 7798.554639] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 7798.558165] Lustre: Skipped 22 previous similar messages [ 7800.289058] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7800.291896] Lustre: Skipped 22 previous similar messages [ 7800.409404] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7801.103194] Lustre: lustre-OST0000: Denying connection for new client 6841a3d2-0418-40cd-80cf-a7acd50aca6f (at 192.168.201.42@tcp), waiting for 3 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 7806.184251] Lustre: lustre-OST0000: Denying connection for new client 6841a3d2-0418-40cd-80cf-a7acd50aca6f (at 192.168.201.42@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 7811.304280] Lustre: lustre-OST0000: Denying connection for new client 6841a3d2-0418-40cd-80cf-a7acd50aca6f (at 192.168.201.42@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7816.424118] Lustre: lustre-OST0000: Denying connection for new client 6841a3d2-0418-40cd-80cf-a7acd50aca6f (at 192.168.201.42@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 7826.663974] Lustre: lustre-OST0000: Denying connection for new client 6841a3d2-0418-40cd-80cf-a7acd50aca6f (at 192.168.201.42@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:43 [ 7826.669234] Lustre: Skipped 1 previous similar message [ 7847.143964] Lustre: lustre-OST0000: Denying connection for new client 6841a3d2-0418-40cd-80cf-a7acd50aca6f (at 192.168.201.42@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:23 [ 7847.150797] Lustre: Skipped 3 previous similar messages [ 7870.500144] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 7870.501729] Lustre: 267709:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client b207b558-d47a-47d1-bebc-de5890f3ef50@ [ 7870.504825] Lustre: 267709:0:(genops.c:1600:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 7870.507464] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 7870.512376] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7870.512442] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 7870.514584] Lustre: Skipped 71 previous similar messages [ 7870.516635] Lustre: Skipped 22 previous similar messages [ 7870.518994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13556 to 0x280000401:13577) [ 7873.609113] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 7882.781683] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 7884.862372] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 01:58:32 (1789192712) [ 7886.258335] Lustre: Failing over lustre-OST0000 [ 7886.300161] Lustre: server umount lustre-OST0000 complete [ 7899.483751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7901.440145] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7905.334128] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 01:58:52 (1789192732) [ 7906.594739] Lustre: Failing over lustre-OST0000 [ 7906.639690] Lustre: server umount lustre-OST0000 complete [ 7919.351563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7921.261733] LustreError: 270950:0:(ldlm_lib.c:2939:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 7921.264067] LustreError: 270950:0:(ldlm_lib.c:2939:target_recovery_thread()) Skipped 77 previous similar messages [ 7921.340750] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7927.263133] Lustre: *** cfs_fail_loc=715, val=40*** [ 7936.991089] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789192748/real 1789192748] req@ffff9f2c0c6b1f80 x1876096409153792/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789192764 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 7937.000887] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 32 previous similar messages [ 7937.003789] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [ 7937.006818] Lustre: Skipped 1 previous similar message [ 7943.135129] Lustre: *** cfs_fail_loc=715, val=40*** [ 7943.136940] Lustre: Skipped 2 previous similar messages [ 7944.159127] Lustre: *** cfs_fail_loc=715, val=40*** [ 7944.161137] Lustre: Skipped 1 previous similar message [ 7952.351657] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:09 [ 7952.354329] Lustre: Skipped 2 previous similar messages [ 7958.495137] Lustre: *** cfs_fail_loc=715, val=40*** [ 7961.303086] LustreError: 270950:0:(ldlm_lib.c:2939:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7961.305216] LustreError: 270950:0:(ldlm_lib.c:2939:target_recovery_thread()) Skipped 77 previous similar messages [ 7963.160332] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7963.662528] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7967.143293] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 01:59:54 (1789192794) [ 7968.454032] Lustre: Failing over lustre-MDT0000 [ 7968.689481] Lustre: server umount lustre-MDT0000 complete [ 7981.409902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7982.754678] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7986.663847] LustreError: 272501:0:(ldlm_lib.c:2939:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 7989.727167] Lustre: *** cfs_fail_loc=715, val=80*** [ 7989.728659] Lustre: Skipped 2 previous similar messages [ 7999.719861] Lustre: lustre-MDT0000: Client 6841a3d2-0418-40cd-80cf-a7acd50aca6f (at 192.168.201.42@tcp) reconnected, waiting for 2 clients in recovery for 1:27 [ 7999.722663] Lustre: Skipped 1 previous similar message [ 8003.039521] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8006.111205] Lustre: *** cfs_fail_loc=715, val=80*** [ 8006.114243] Lustre: Skipped 1 previous similar message [ 8016.104571] Lustre: lustre-MDT0000: Client 6841a3d2-0418-40cd-80cf-a7acd50aca6f (at 192.168.201.42@tcp) reconnected, waiting for 2 clients in recovery for 1:11 [ 8018.400077] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8022.495193] Lustre: *** cfs_fail_loc=715, val=80*** [ 8022.497884] Lustre: Skipped 1 previous similar message [ 8031.463990] Lustre: lustre-MDT0000: Client 6841a3d2-0418-40cd-80cf-a7acd50aca6f (at 192.168.201.42@tcp) reconnected, waiting for 2 clients in recovery for 0:56 [ 8034.783580] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8047.847887] Lustre: lustre-MDT0000: Client 6841a3d2-0418-40cd-80cf-a7acd50aca6f (at 192.168.201.42@tcp) reconnected, waiting for 2 clients in recovery for 0:39 [ 8051.167410] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8057.311143] Lustre: *** cfs_fail_loc=715, val=80*** [ 8057.312613] Lustre: Skipped 4 previous similar messages [ 8064.232177] Lustre: lustre-MDT0000: Client 6841a3d2-0418-40cd-80cf-a7acd50aca6f (at 192.168.201.42@tcp) reconnected, waiting for 2 clients in recovery for 0:23 [ 8066.527766] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8066.743103] LustreError: 272501:0:(ldlm_lib.c:2939:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8066.749610] Lustre: 272501:0:(ldlm_lib.c:2985:target_recovery_thread()) too long recovery - read logs [ 8066.753068] LustreError: dumping log to /tmp/lustre-log.1789192894.272501 [ 8066.814356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13591 to 0x280000401:13609) [ 8066.814357] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13550 to 0x2c0000401:13569) [ 8068.628468] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8069.114345] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8072.596947] Lustre: DEBUG MARKER: == replay-single test complete, duration 7792 sec ======== 02:01:39 (1789192899) [ 8073.118776] Lustre: DEBUG MARKER: === replay-single: start cleanup 02:01:40 (1789192900) === [ 8075.537265] Lustre: DEBUG MARKER: === replay-single: finish cleanup 02:01:42 (1789192902) === [ 8112.023864] Lustre: server umount lustre-MDT0000 complete [ 8114.737736] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789192942 with bad export cookie 16708211896561840195 [ 8114.740778] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8114.852888] Lustre: server umount lustre-MDT0001 complete [ 8128.004486] Lustre: server umount lustre-OST0000 complete [ 8140.402321] Lustre: server umount lustre-OST0001 complete [ 8146.236164] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 8147.563445] Key type lgssc unregistered [ 8147.732815] LNet: 275803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8147.737892] LNetError: 275803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8147.748712] LNet: Removed LNI 192.168.201.142@tcp [ 8148.262242] Key type .llcrypt unregistered [ 8148.264595] Key type ._llcrypt unregistered