[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 502574534 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002444] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.007467] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.011104] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.014032] SELinux: Initializing. [ 0.015075] *** VALIDATE selinux *** [ 0.023641] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028280] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029166] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031053] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032110] *** VALIDATE tmpfs *** [ 0.034053] *** VALIDATE proc *** [ 0.035041] *** VALIDATE cgroup *** [ 0.035881] *** VALIDATE cgroup2 *** [ 0.037050] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040021] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.043750] debug: unmapping init [mem 0xffffffff86e59000-0xffffffff86e60fff] [ 0.045197] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047021] ... version: 2 [ 0.047965] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.048933] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.049969] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051319] rcu: Hierarchical SRCU implementation. [ 0.053512] smp: Bringing up secondary CPUs ... [ 0.054647] x86: Booting SMP configuration: [ 0.055026] .... node #0, CPUs: #1 #2 #3 [ 0.058363] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138214] node 0 deferred pages initialised in 74ms [ 0.141008] devtmpfs: initialized [ 0.143242] x86/mm: Memory block size: 128MB [ 0.148110] gcov: version magic: 0x41383552 [ 0.150379] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159634] pinctrl core: initialized pinctrl subsystem [ 0.163206] [ 0.164007] ************************************************************* [ 0.167014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171014] ** ** [ 0.175015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178011] ** ** [ 0.181021] ** This means that this kernel is built to expose internal ** [ 0.184012] ** IOMMU data structures, which may compromise security on ** [ 0.188013] ** your system. ** [ 0.191014] ** ** [ 0.194012] ** If you see this message and you are not debugging the ** [ 0.197014] ** kernel, report this immediately to your vendor! ** [ 0.200011] ** ** [ 0.203012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.206012] ************************************************************* [ 0.209604] NET: Registered protocol family 16 [ 0.211337] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.213048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.216063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.220011] cpuidle: using governor menu [ 0.221725] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.222550] PCI: Using configuration type 1 for base access [ 0.223141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.233048] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.236020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.240120] cryptd: max_cpu_qlen set to 1000 [ 0.244254] ACPI: Added _OSI(Module Device) [ 0.247016] ACPI: Added _OSI(Processor Device) [ 0.248013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.256409] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.263571] ACPI: Interpreter enabled [ 0.264050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.266011] ACPI: Using IOAPIC for interrupt routing [ 0.267103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270405] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.280979] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.283042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.286015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.289077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.293357] acpiphp: Slot [2] registered [ 0.295141] acpiphp: Slot [5] registered [ 0.296128] acpiphp: Slot [6] registered [ 0.298121] acpiphp: Slot [7] registered [ 0.299086] acpiphp: Slot [8] registered [ 0.301103] acpiphp: Slot [9] registered [ 0.302093] acpiphp: Slot [10] registered [ 0.304122] acpiphp: Slot [3] registered [ 0.305089] acpiphp: Slot [4] registered [ 0.306134] acpiphp: Slot [11] registered [ 0.308118] acpiphp: Slot [12] registered [ 0.310148] acpiphp: Slot [13] registered [ 0.311072] acpiphp: Slot [14] registered [ 0.313103] acpiphp: Slot [15] registered [ 0.314159] acpiphp: Slot [16] registered [ 0.315096] acpiphp: Slot [17] registered [ 0.317096] acpiphp: Slot [18] registered [ 0.318085] acpiphp: Slot [19] registered [ 0.320114] acpiphp: Slot [20] registered [ 0.321090] acpiphp: Slot [21] registered [ 0.322078] acpiphp: Slot [22] registered [ 0.324181] acpiphp: Slot [23] registered [ 0.325085] acpiphp: Slot [24] registered [ 0.326080] acpiphp: Slot [25] registered [ 0.328125] acpiphp: Slot [26] registered [ 0.329087] acpiphp: Slot [27] registered [ 0.330149] acpiphp: Slot [28] registered [ 0.332090] acpiphp: Slot [29] registered [ 0.333091] acpiphp: Slot [30] registered [ 0.335102] acpiphp: Slot [31] registered [ 0.336052] PCI host bridge to bus 0000:00 [ 0.338025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.341026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.343022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.346022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.348017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.350023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.352194] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.356478] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.358981] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.368014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.373933] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.382018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.385091] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.387866] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.391045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.394963] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.401015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.415015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.420037] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.427432] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.443017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.454026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.495018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.518108] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.525017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.531016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.555019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.569240] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.579022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.590016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.619024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.637708] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.649017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.668025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.689014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.699845] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.716020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.734018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.756017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.774281] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.800020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.821019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.870025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.882323] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.885353] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.888438] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.891329] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.893386] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.900127] iommu: Default domain type: Passthrough [ 0.902417] SCSI subsystem initialized [ 0.903137] ACPI: bus type USB registered [ 0.905111] usbcore: registered new interface driver usbfs [ 0.907126] usbcore: registered new interface driver hub [ 0.908066] usbcore: registered new device driver usb [ 0.910163] pps_core: LinuxPPS API ver. 1 registered [ 0.912009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.915060] PTP clock support registered [ 0.917044] EDAC MC: Ver: 3.0.0 [ 0.918452] PCI: Using ACPI for IRQ routing [ 0.920891] NetLabel: Initializing [ 0.922010] NetLabel: domain hash size = 128 [ 0.923008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.925084] NetLabel: unlabeled traffic allowed by default [ 0.927152] vgaarb: loaded [ 0.928257] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.931013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.938135] clocksource: Switched to clocksource kvm-clock [ 1.045488] VFS: Disk quotas dquot_6.6.0 [ 1.047462] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.050495] *** VALIDATE ramfs *** [ 1.052030] *** VALIDATE hugetlbfs *** [ 1.053920] pnp: PnP ACPI init [ 1.056986] pnp: PnP ACPI: found 6 devices [ 1.073917] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.077398] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.079526] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.081675] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.084056] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.086327] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.088880] NET: Registered protocol family 2 [ 1.091285] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.096265] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.099599] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.104819] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.108870] TCP: Hash tables configured (established 65536 bind 65536) [ 1.111575] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.114549] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.117340] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.120034] NET: Registered protocol family 1 [ 1.123085] RPC: Registered named UNIX socket transport module. [ 1.125099] RPC: Registered udp transport module. [ 1.126568] RPC: Registered tcp transport module. [ 1.128174] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.130410] NET: Registered protocol family 44 [ 1.131580] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.133138] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.135280] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.138160] PCI: CLS 0 bytes, default 64 [ 1.139697] Unpacking initramfs... [ 2.498569] debug: unmapping init [mem 0xffff9595fcc54000-0xffff9595fffbffff] [ 2.502400] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.504041] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.506021] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.074617] Initialise system trusted keyrings [ 3.076365] Key type blacklist registered [ 3.077632] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.085701] zbud: loaded [ 3.088155] *** VALIDATE nfs *** [ 3.088943] *** VALIDATE nfs4 *** [ 3.090027] pstore: using deflate compression [ 3.093366] Platform Keyring initialized [ 3.200404] NET: Registered protocol family 38 [ 3.202178] Key type asymmetric registered [ 3.204145] Asymmetric key parser 'x509' registered [ 3.206197] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.208891] io scheduler mq-deadline registered [ 3.210565] io scheduler kyber registered [ 3.212218] io scheduler bfq registered [ 3.213987] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.216526] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.219348] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.222079] ACPI: Power Button [PWRF] [ 3.226823] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.232507] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.246176] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.253801] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.278702] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.309747] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.340288] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.345455] Non-volatile memory driver v1.3 [ 3.347120] Linux agpgart interface v0.103 [ 3.378642] virtio_blk virtio1: [vda] 145808 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.381790] vda: detected capacity change from 0 to 74653696 [ 3.395810] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.398740] vdb: detected capacity change from 0 to 1073741824 [ 3.413020] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.416097] vdc: detected capacity change from 0 to 2621440000 [ 3.432011] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.434449] vdd: detected capacity change from 0 to 2621440000 [ 3.448525] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.452193] vde: detected capacity change from 0 to 4294967296 [ 3.468596] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.472053] vdf: detected capacity change from 0 to 4294967296 [ 3.478329] libphy: Fixed MDIO Bus: probed [ 3.483608] usbcore: registered new interface driver usbserial_generic [ 3.486186] usbserial: USB Serial support registered for generic [ 3.488739] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.494337] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.496271] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.498756] mousedev: PS/2 mouse device common for all mice [ 3.501900] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.505812] rtc_cmos 00:05: RTC can wake from S4 [ 3.510787] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.515363] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.519226] rtc_cmos 00:05: registered as rtc0 [ 3.526181] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.529221] intel_pstate: CPU model not supported [ 3.531800] hid: raw HID events driver (C) Jiri Kosina [ 3.533844] usbcore: registered new interface driver usbhid [ 3.535420] usbhid: USB HID core driver [ 3.536812] drop_monitor: Initializing network drop monitor service [ 3.539435] Initializing XFRM netlink socket [ 3.541625] NET: Registered protocol family 10 [ 3.544312] Segment Routing with IPv6 [ 3.545752] NET: Registered protocol family 17 [ 3.547682] mpls_gso: MPLS GSO support [ 3.553032] RAS: Correctable Errors collector initialized. [ 3.554891] AVX version of gcm_enc/dec engaged. [ 3.556361] AES CTR mode by8 optimization enabled [ 3.631981] sched_clock: Marking stable (3631961120, 0)->(4555035917, -923074797) [ 3.635862] registered taskstats version 1 [ 3.637996] Loading compiled-in X.509 certificates [ 3.640314] zswap: loaded using pool lzo/zbud [ 3.662859] Key type big_key registered [ 3.674773] Key type encrypted registered [ 3.676510] ima: No TPM chip found, activating TPM-bypass! [ 3.678524] ima: Allocated hash algorithm: sha1 [ 3.680332] ima: No architecture policies found [ 3.682168] evm: Initialising EVM extended attributes: [ 3.683879] evm: security.selinux [ 3.684871] evm: security.ima [ 3.685643] evm: security.capability [ 3.686798] evm: HMAC attrs: 0x1 [ 3.689187] rtc_cmos 00:05: setting system clock to 2026-07-13 12:40:00 UTC (1783946400) [ 3.694527] debug: unmapping init [mem 0xffffffff87e03000-0xffffffff87ffffff] [ 3.698135] debug: unmapping init [mem 0xffffffff86b82000-0xffffffff86e58fff] [ 3.706089] Write protecting the kernel read-only data: 28672k [ 3.709313] debug: unmapping init [mem 0xffffffff85203000-0xffffffff853fffff] [ 3.712065] debug: unmapping init [mem 0xffffffff85b14000-0xffffffff85bfffff] [ 3.745969] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.753196] systemd[1]: Detected virtualization kvm. [ 3.754723] systemd[1]: Detected architecture x86-64. [ 3.756163] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.786067] systemd[1]: No hostname configured. [ 3.787447] systemd[1]: Set hostname to . [ 3.789483] random: systemd: uninitialized urandom read (16 bytes read) [ 3.792080] systemd[1]: Initializing machine ID from random generator. [ 3.858155] random: ln: uninitialized urandom read (6 bytes read) [ 3.958154] random: systemd: uninitialized urandom read (16 bytes read) [ 3.961329] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.966987] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.971800] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.728515] device-mapper: uevent: version 1.0.3 [ 4.731223] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.493288] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.518943] random: fast init done [ 5.667116] scsi host0: ata_piix [ 5.701117] scsi host1: ata_piix [ 5.703377] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.707948] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.353030] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.191346] random: crng init done [ 10.195205] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.708703] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.856666] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.087689] SELinux: Disabled at runtime. [ 12.144443] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.150301] systemd[1]: Detected virtualization kvm. [ 12.151468] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.612229] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.614607] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.619385] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.621676] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.623566] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.628429] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.631120] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ 12.659538] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.049630] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.318313] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.345978] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.455891] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.465293] EDAC sbridge: Ver: 1.1.2 [ 14.860483] Key type dns_resolver registered [ 15.140550] NFS: Registering the id_resolver key type [ 15.142341] Key type id_resolver registered [ 15.143308] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg213-server login: [ 103.297078] hrtimer: interrupt took 5204703 ns [ 108.738333] libcfs: loading out-of-tree module taints kernel. [ 108.819767] Key type ._llcrypt registered [ 108.821696] Key type .llcrypt registered [ 108.997540] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 128.345112] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 129.864417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 129.898705] alg: No test for adler32 (adler32-zlib) [ 131.222410] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 131.906901] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 133.600197] Key type lgssc registered [ 134.984422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 151.587673] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 198.923702] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 215.801146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 215.854862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.069170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 217.103384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 217.298178] Lustre: lustre-MDT0000: new disk, initializing [ 217.367339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.400602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 224.443670] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.949234] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 244.673080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.129600] Lustre: lustre-OST0000: new disk, initializing [ 245.142208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.152376] Lustre: Skipped 1 previous similar message [ 245.158196] Lustre: 6985:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.244149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.707779] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.548246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 251.561391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 251.591402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 267.520280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.710908] Lustre: lustre-OST0001: new disk, initializing [ 267.716908] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 267.722561] Lustre: 8051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 267.815429] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 274.550610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 274.559808] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 274.597636] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 275.675471] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 289.456696] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 301.038582] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 308.433434] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing check_logdir /tmp/testlogs/ [ 319.188295] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing yml_node [ 325.694846] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 330.356595] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 333.802320] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 335.947090] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Mon Jul 13 08:45:30 EDT 2026 [ 357.068519] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 359.223864] Lustre: DEBUG MARKER: === replay-single: start setup 08:45:53 (1783946753) === [ 366.394290] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing check_config_client /mnt/lustre [ 385.306378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 389.567178] Lustre: 11703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 393.951813] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 399.241790] Lustre: DEBUG MARKER: === replay-single: finish setup 08:46:34 (1783946794) === [ 401.397802] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 08:46:36 (1783946796) [ 410.407776] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 413.290399] Lustre: Failing over lustre-MDT0000 [ 414.228618] Lustre: server umount lustre-MDT0000 complete [ 434.144125] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783946814/real 1783946814] req@ffff95967e4aca80 x1870603514606208/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783946830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 434.144293] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.163689] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 434.172499] Lustre: Skipped 1 previous similar message [ 434.182223] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 438.304143] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783946819/real 1783946819] req@ffff95967e48ed80 x1870603514606848/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783946835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 439.750917] LDISKFS-fs (dm-0): 9 truncates cleaned up [ 439.755857] LDISKFS-fs (dm-0): recovery complete [ 439.765703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.361250] Lustre: 3449:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783946824/real 1783946824] req@ffff95967e178380 x1870603514607104/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783946840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 443.406210] Lustre: 3449:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 444.384703] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95967e4aed80 x1870603514608512/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 444.797816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.800844] Lustre: 3450:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783946829/real 1783946829] req@ffff95967e442d80 x1870603514607488/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783946845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 448.834772] Lustre: 3450:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 450.651507] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 458.590168] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 458.691413] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 459.055541] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 464.616695] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 465.691533] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 488.169813] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 08:48:01 (1783946881) [ 494.455456] Lustre: Failing over lustre-OST0000 [ 494.562145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 494.584583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.631981] Lustre: server umount lustre-OST0000 complete [ 500.214243] LustreError: 6972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.228312] LustreError: 6972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 505.321943] LustreError: 8870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.443173] LustreError: 6969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.566149] LustreError: 6972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.071882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 518.440893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 519.976456] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 520.016279] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 520.028297] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 520.066686] Lustre: Skipped 1 previous similar message [ 526.433587] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 537.270812] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 539.246449] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 550.982837] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 08:49:05 (1783946945) [ 560.693473] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 563.564546] Lustre: Failing over lustre-MDT0000 [ 563.956915] Lustre: server umount lustre-MDT0000 complete [ 581.088673] Lustre: 3450:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783946961/real 1783946961] req@ffff95967ec5c700 x1870603514643968/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783946977 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 581.089193] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.130145] Lustre: 3450:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 581.130325] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 581.169618] Lustre: Skipped 1 previous similar message [ 590.304300] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783946971/real 1783946971] req@ffff959644bcc700 x1870603514644736/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783946987 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 590.364037] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 591.248877] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 591.257575] LDISKFS-fs (dm-0): recovery complete [ 591.273836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.665167] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95967ec5dc00 x1870603514646144/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 606.327521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 606.389557] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 610.336453] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 612.565516] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 612.586629] Lustre: lustre-MDT0000: Denying connection for new client 1d820d32-e553-4611-a472-2ac7c0406bb1 (at 192.168.202.13@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 617.822897] Lustre: lustre-MDT0000: Denying connection for new client 1d820d32-e553-4611-a472-2ac7c0406bb1 (at 192.168.202.13@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 622.945793] Lustre: lustre-MDT0000: Denying connection for new client 1d820d32-e553-4611-a472-2ac7c0406bb1 (at 192.168.202.13@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 625.699467] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 628.059789] Lustre: lustre-MDT0000: Denying connection for new client 1d820d32-e553-4611-a472-2ac7c0406bb1 (at 192.168.202.13@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:44 [ 633.187219] Lustre: lustre-MDT0000: Denying connection for new client 1d820d32-e553-4611-a472-2ac7c0406bb1 (at 192.168.202.13@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 643.428210] Lustre: lustre-MDT0000: Denying connection for new client 1d820d32-e553-4611-a472-2ac7c0406bb1 (at 192.168.202.13@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:29 [ 643.450597] Lustre: Skipped 1 previous similar message [ 663.900787] Lustre: lustre-MDT0000: Denying connection for new client 1d820d32-e553-4611-a472-2ac7c0406bb1 (at 192.168.202.13@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:08 [ 663.917657] Lustre: Skipped 3 previous similar messages [ 672.501291] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 672.511128] Lustre: 16649:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 4189f744-bfd8-4ddb-8113-9c5a4d972cdf@ [ 672.530271] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 672.561779] Lustre: lustre-MDT0000: Recovery over after 1:00, of 1 clients 0 recovered and 1 was evicted. [ 672.638248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:44 to 0x240000400:65) [ 672.639863] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:65) [ 684.248282] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 08:51:19 (1783947079) [ 693.339415] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 696.334239] Lustre: Failing over lustre-MDT0000 [ 696.885257] Lustre: server umount lustre-MDT0000 complete [ 713.760305] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783947094/real 1783947094] req@ffff95967e4adf80 x1870603514671104/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783947110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 713.788791] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 713.790141] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 713.795592] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.043817] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 722.045244] LDISKFS-fs (dm-0): recovery complete [ 722.050311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.424858] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9596447dd880 x1870603514672896/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 724.115538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 724.169640] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 729.707161] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 732.730332] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 732.739887] Lustre: lustre-MDT0000: Denying connection for new client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 732.767620] Lustre: Skipped 1 previous similar message [ 738.348829] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 738.359219] Lustre: Skipped 1 previous similar message [ 792.502144] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 792.517619] Lustre: 18386:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 1d820d32-e553-4611-a472-2ac7c0406bb1@ [ 792.531060] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 792.578331] Lustre: lustre-MDT0000: Recovery over after 1:00, of 1 clients 0 recovered and 1 was evicted. [ 792.675663] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:97) [ 792.677407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:44 to 0x240000400:97) [ 806.636662] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 08:53:21 (1783947201) [ 817.471405] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 820.487786] Lustre: Failing over lustre-MDT0000 [ 821.058856] Lustre: server umount lustre-MDT0000 complete [ 841.697270] Lustre: 3450:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783947222/real 1783947222] req@ffff95967e580e00 x1870603514697472/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783947238 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 841.698947] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.700305] Lustre: Skipped 1 previous similar message [ 841.705835] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 841.717744] Lustre: 3450:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 845.318068] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 845.325918] LDISKFS-fs (dm-0): recovery complete [ 845.348525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.310931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 852.379395] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 857.528337] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 866.538627] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 866.546898] Lustre: Skipped 1 previous similar message [ 867.170374] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 867.327711] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 867.412475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:44 to 0x240000400:129) [ 867.414442] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:129) [ 874.674620] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 876.919492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 887.712138] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 08:54:42 (1783947282) [ 898.160835] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 900.610671] Lustre: Failing over lustre-MDT0000 [ 901.144230] Lustre: server umount lustre-MDT0000 complete [ 919.009185] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783947299/real 1783947299] req@ffff9596656b1500 x1870603514716544/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783947315 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 919.022717] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 919.051407] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 919.051491] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.051494] Lustre: Skipped 1 previous similar message [ 926.764827] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 926.772482] LDISKFS-fs (dm-0): recovery complete [ 926.783869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.737177] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9596675d8000 x1870603514718208/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 929.228826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.276748] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 935.265153] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 943.490175] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 943.508220] Lustre: Skipped 1 previous similar message [ 943.968519] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 944.181514] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 944.267133] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 944.269333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:44 to 0x240000400:161) [ 951.675420] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 953.144298] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 963.150750] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 08:55:58 (1783947358) [ 973.374407] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 975.888358] Lustre: Failing over lustre-MDT0000 [ 976.301336] Lustre: server umount lustre-MDT0000 complete [ 995.808150] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 995.826229] Lustre: Skipped 2 previous similar messages [ 995.830944] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1001.725972] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1001.731903] LDISKFS-fs (dm-0): recovery complete [ 1001.744681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.059605] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba441bdd53 [ 1006.092424] Lustre: MGC192.168.202.113@tcp: Connection restored to 0@lo (at 0@lo) [ 1006.100819] Lustre: Skipped 1 previous similar message [ 1006.581949] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1008.243969] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1008.408103] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1008.468921] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 1008.470084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 1012.051575] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1023.097513] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1024.831071] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1035.696916] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 08:57:09 (1783947429) [ 1044.938824] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1047.531485] Lustre: Failing over lustre-MDT0000 [ 1048.183787] Lustre: server umount lustre-MDT0000 complete [ 1068.000989] Lustre: 3449:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783947448/real 1783947448] req@ffff95967e52b480 x1870603514754432/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783947464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1068.009299] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.045319] Lustre: 3449:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 17 previous similar messages [ 1068.071806] Lustre: Skipped 1 previous similar message [ 1068.074500] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1074.987989] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1074.990284] LDISKFS-fs (dm-0): recovery complete [ 1075.008800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.233412] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba441be223 [ 1077.264990] Lustre: MGC192.168.202.113@tcp: Connection restored to 0@lo (at 0@lo) [ 1077.275406] Lustre: Skipped 2 previous similar messages [ 1077.598803] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (not set up) [ 1077.976949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1077.984916] Lustre: Skipped 1 previous similar message [ 1078.021324] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1079.024405] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1079.185918] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1079.278618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:225) [ 1079.283270] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 1083.679251] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1097.998322] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1100.572519] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1111.327744] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 08:58:25 (1783947505) [ 1120.867632] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1123.200528] Lustre: Failing over lustre-MDT0000 [ 1123.299077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.304549] Lustre: Skipped 1 previous similar message [ 1123.745977] Lustre: server umount lustre-MDT0000 complete [ 1144.800430] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1147.108761] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1147.120902] LDISKFS-fs (dm-0): recovery complete [ 1147.153873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.987279] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95967e529880 x1870603514774272/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1155.579809] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1160.423239] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1163.775310] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1163.783398] Lustre: Skipped 2 previous similar messages [ 1164.640379] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1164.765438] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1164.813935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:257) [ 1164.813935] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:257) [ 1171.587835] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1173.337362] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1181.792106] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 08:59:36 (1783947576) [ 1182.712268] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1182.716641] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1182.724358] LustreError: 27910:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95967d879180 x1870603496848256/t38654705666(0) o35->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:376/0 lens 392/456 e 0 to 0 dl 1783947596 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1191.691373] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1193.319349] Lustre: Failing over lustre-MDT0000 [ 1193.517279] Lustre: server umount lustre-MDT0000 complete [ 1210.660156] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.688195] Lustre: Skipped 2 previous similar messages [ 1210.774513] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1219.553908] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1219.556073] LDISKFS-fs (dm-0): recovery complete [ 1219.573625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.832472] LustreError: 29892:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1220.843486] LustreError: 29892:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9596656b1180 x1870603514791424/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1783947617 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1220.871361] LustreError: 29892:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1221.090047] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95967d87df80 x1870603514791680/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1221.774583] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1227.458670] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1230.268659] Lustre: 29907:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95967d879500 x1870603496848256/t38654705666(0) o35->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:424/0 lens 392/456 e 0 to 0 dl 1783947644 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1230.316315] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:289) [ 1230.322215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:289) [ 1239.079374] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1240.830911] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1250.095444] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 09:00:44 (1783947644) [ 1259.039555] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1261.340123] Lustre: Failing over lustre-MDT0000 [ 1261.681255] Lustre: server umount lustre-MDT0000 complete [ 1286.707606] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1286.711355] LDISKFS-fs (dm-0): recovery complete [ 1286.727730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.431818] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba441bef97 [ 1291.628447] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (not set up) [ 1291.778860] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1292.772680] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1292.783289] Lustre: Skipped 4 previous similar messages [ 1293.032545] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1293.035428] Lustre: Skipped 1 previous similar message [ 1293.088690] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1293.097282] Lustre: Skipped 1 previous similar message [ 1293.139319] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:321) [ 1293.142732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:321) [ 1297.017195] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1307.717598] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1309.710915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1320.429912] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 09:01:55 (1783947715) [ 1329.695930] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1331.195965] Lustre: *** cfs_fail_loc=114, val=0*** [ 1335.247312] Lustre: Failing over lustre-MDT0000 [ 1335.548609] Lustre: server umount lustre-MDT0000 complete [ 1354.208198] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783947735/real 1783947735] req@ffff9596447dc700 x1870603514825856/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783947751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1354.241720] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [ 1354.249842] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.277497] Lustre: Skipped 3 previous similar messages [ 1355.170986] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1355.212385] LustreError: Skipped 1 previous similar message [ 1360.822968] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1360.837876] LDISKFS-fs (dm-0): recovery complete [ 1360.868159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.410985] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9596656b1500 x1870603514827776/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1365.817771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.824705] Lustre: Skipped 3 previous similar messages [ 1371.546346] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1378.777114] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:353) [ 1378.791517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:353) [ 1385.889993] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1387.484855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1397.362538] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 09:03:12 (1783947792) [ 1406.426298] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1407.550711] Lustre: *** cfs_fail_loc=128, val=0*** [ 1411.120548] Lustre: Failing over lustre-MDT0000 [ 1411.388775] Lustre: server umount lustre-MDT0000 complete [ 1436.381664] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1436.386257] LDISKFS-fs (dm-0): recovery complete [ 1436.393685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.198980] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1442.206281] Lustre: Skipped 1 previous similar message [ 1447.428606] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1455.560058] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:385) [ 1455.562705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:385) [ 1461.414885] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1463.294269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1473.848617] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 09:04:28 (1783947868) [ 1481.569513] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1483.688027] Lustre: Failing over lustre-MDT0000 [ 1483.968729] Lustre: server umount lustre-MDT0000 complete [ 1506.596566] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1506.600070] LDISKFS-fs (dm-0): recovery complete [ 1506.610555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1513.955170] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba441bff0a [ 1519.361286] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1527.959426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:391 to 0x240000400:417) [ 1527.965199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:391 to 0x280000400:417) [ 1534.603948] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1536.271826] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1546.197569] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 09:05:41 (1783947941) [ 1555.142344] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1557.446247] Lustre: Failing over lustre-MDT0000 [ 1557.644976] LustreError: 39229:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9596676b5e00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1557.874608] Lustre: server umount lustre-MDT0000 complete [ 1581.217990] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1581.221549] LDISKFS-fs (dm-0): recovery complete [ 1581.228280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1586.912738] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95967e52bb80 x1870603514882304/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1592.161174] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1594.208282] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1594.216791] Lustre: Skipped 3 previous similar messages [ 1594.420667] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1594.427151] Lustre: Skipped 3 previous similar messages [ 1594.472568] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:423 to 0x280000400:449) [ 1594.473959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:423 to 0x240000400:449) [ 1600.612987] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1600.625139] Lustre: Skipped 8 previous similar messages [ 1603.954515] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1606.079768] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1616.555153] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 09:06:51 (1783948011) [ 1624.682756] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1633.495595] Lustre: Failing over lustre-MDT0000 [ 1633.979405] Lustre: server umount lustre-MDT0000 complete [ 1652.200780] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.201386] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.214391] LustreError: Skipped 3 previous similar messages [ 1652.242834] Lustre: Skipped 7 previous similar messages [ 1657.274664] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1657.277625] LDISKFS-fs (dm-0): recovery complete [ 1657.284984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.432864] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff959667bdb800 x1870603514904064/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1667.141901] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1680.078079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:577) [ 1680.078818] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:577) [ 1687.531943] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1689.232629] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1720.887918] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 09:08:35 (1783948115) [ 1730.462621] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1732.656266] Lustre: Failing over lustre-MDT0000 [ 1733.210771] Lustre: server umount lustre-MDT0000 complete [ 1758.269567] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1758.272118] LDISKFS-fs (dm-0): recovery complete [ 1758.285930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.725655] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1760.733899] Lustre: Skipped 3 previous similar messages [ 1763.840455] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:609) [ 1763.842177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:609) [ 1765.966462] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1777.400720] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1779.172736] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1791.128806] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 09:09:45 (1783948185) [ 1800.886348] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1803.694969] Lustre: Failing over lustre-MDT0000 [ 1804.401060] Lustre: server umount lustre-MDT0000 complete [ 1829.631595] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1829.636584] LDISKFS-fs (dm-0): recovery complete [ 1829.657089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1839.363857] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1845.816298] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:641) [ 1845.830495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:641) [ 1853.659334] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1856.382470] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1867.551700] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 09:11:01 (1783948261) [ 1877.712700] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1880.682246] Lustre: Failing over lustre-MDT0000 [ 1881.451531] Lustre: server umount lustre-MDT0000 complete [ 1899.296525] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783948280/real 1783948280] req@ffff95967d878380 x1870603515019136/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783948296 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1899.354939] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 63 previous similar messages [ 1908.264284] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1908.271324] LDISKFS-fs (dm-0): recovery complete [ 1908.289741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.472119] LustreError: 47721:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1909.487765] LustreError: 47721:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff95964faa8a80 x1870603515020160/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1783948306 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1909.518754] LustreError: 47721:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1910.125494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.130388] Lustre: Skipped 6 previous similar messages [ 1915.384910] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1922.611056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:673) [ 1922.614246] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:673) [ 1930.720650] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1932.494852] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1942.852767] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 09:12:17 (1783948337) [ 1952.112482] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1954.318577] Lustre: Failing over lustre-MDT0000 [ 1954.724180] Lustre: server umount lustre-MDT0000 complete [ 1978.238897] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1978.247613] LDISKFS-fs (dm-0): recovery complete [ 1978.259926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1984.047338] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:705) [ 1984.051363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:705) [ 1987.284833] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1998.455336] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2000.181871] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2010.437415] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 09:13:25 (1783948405) [ 2018.497672] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2020.991413] Lustre: Failing over lustre-MDT0000 [ 2021.517509] Lustre: server umount lustre-MDT0000 complete [ 2045.251602] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2045.254335] LDISKFS-fs (dm-0): recovery complete [ 2045.261420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.677509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:737) [ 2051.681177] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:737) [ 2055.951598] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2066.577623] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2069.183307] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2078.140495] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 09:14:33 (1783948473) [ 2087.594944] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2090.245560] Lustre: Failing over lustre-MDT0000 [ 2090.692513] Lustre: server umount lustre-MDT0000 complete [ 2115.770209] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2115.772641] LDISKFS-fs (dm-0): recovery complete [ 2115.778581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.185032] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff959647483b80 x1870603515074688/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2126.497624] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2133.351292] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2133.368042] Lustre: Skipped 6 previous similar messages [ 2133.482322] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2133.491827] Lustre: Skipped 6 previous similar messages [ 2133.578663] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:769) [ 2133.584420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:769) [ 2137.059242] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2137.067728] Lustre: Skipped 13 previous similar messages [ 2140.178404] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2142.058910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2151.403248] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 09:15:46 (1783948546) [ 2161.783931] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2164.200739] Lustre: Failing over lustre-MDT0000 [ 2164.493566] Lustre: server umount lustre-MDT0000 complete [ 2184.160833] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.162386] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2184.178372] Lustre: Skipped 13 previous similar messages [ 2184.198272] LustreError: Skipped 6 previous similar messages [ 2190.638665] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2190.640728] LDISKFS-fs (dm-0): recovery complete [ 2190.647242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.411602] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba441cc3dc [ 2196.006800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:771 to 0x240000400:801) [ 2196.008518] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:771 to 0x280000400:801) [ 2201.276729] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2213.157624] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2215.075419] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2224.986958] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 09:16:59 (1783948619) [ 2234.857882] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2237.246169] Lustre: Failing over lustre-MDT0000 [ 2237.680601] Lustre: server umount lustre-MDT0000 complete [ 2262.632847] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2262.635022] LDISKFS-fs (dm-0): recovery complete [ 2262.645646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.570738] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba441cc8dd [ 2268.723750] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:771 to 0x280000400:833) [ 2268.732407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:771 to 0x240000400:833) [ 2271.902987] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2282.748455] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2284.552234] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2294.225828] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 09:18:09 (1783948689) [ 2304.114549] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2306.729508] Lustre: Failing over lustre-MDT0000 [ 2307.049728] Lustre: server umount lustre-MDT0000 complete [ 2330.626247] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2330.632726] LDISKFS-fs (dm-0): recovery complete [ 2330.649927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.126790] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2337.144429] Lustre: Skipped 7 previous similar messages [ 2343.447688] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2350.650324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:771 to 0x240000400:865) [ 2350.653902] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:835 to 0x280000400:865) [ 2358.274848] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2360.325049] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2370.678941] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 09:19:25 (1783948765) [ 2378.543663] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2380.546430] Lustre: Failing over lustre-MDT0000 [ 2380.852076] Lustre: server umount lustre-MDT0000 complete [ 2404.652827] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2404.658225] LDISKFS-fs (dm-0): recovery complete [ 2404.689442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2409.891199] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff959646217b80 x1870603515147264/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2416.306361] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2422.297553] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:835 to 0x280000400:897) [ 2422.304264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:867 to 0x240000400:897) [ 2429.203388] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2431.485674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2439.937136] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 09:20:35 (1783948835) [ 2449.881616] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2452.102200] Lustre: Failing over lustre-MDT0000 [ 2452.455673] Lustre: server umount lustre-MDT0000 complete [ 2476.715412] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2476.719571] LDISKFS-fs (dm-0): recovery complete [ 2476.738983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.990969] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2494.075356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:899 to 0x240000400:929) [ 2494.078797] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:899 to 0x280000400:929) [ 2500.101737] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2501.797649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2511.708104] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 09:21:46 (1783948906) [ 2521.812544] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2524.699613] Lustre: Failing over lustre-MDT0000 [ 2525.125536] Lustre: server umount lustre-MDT0000 complete [ 2543.072099] Lustre: 3449:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783948923/real 1783948923] req@ffff959646214e00 x1870603515181824/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783948939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2543.110246] Lustre: 3449:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 85 previous similar messages [ 2552.367196] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2552.374929] LDISKFS-fs (dm-0): recovery complete [ 2552.396476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.169097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.176836] Lustre: Skipped 8 previous similar messages [ 2570.855662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:931 to 0x240000400:961) [ 2570.860995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:899 to 0x280000400:961) [ 2573.806818] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2584.465250] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2586.309518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2595.804663] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 09:23:10 (1783948990) [ 2604.140677] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2606.833293] Lustre: Failing over lustre-MDT0000 [ 2607.105905] Lustre: server umount lustre-MDT0000 complete [ 2630.809225] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2630.812361] LDISKFS-fs (dm-0): recovery complete [ 2630.819553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.291577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:931 to 0x240000400:993) [ 2637.298986] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:963 to 0x280000400:993) [ 2639.608079] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2650.186558] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2651.778828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2661.758993] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 09:24:16 (1783949056) [ 2670.180977] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2673.048566] Lustre: Failing over lustre-MDT0000 [ 2673.272354] Lustre: server umount lustre-MDT0000 complete [ 2697.003662] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2697.008710] LDISKFS-fs (dm-0): recovery complete [ 2697.018879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.233409] LustreError: 69175:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2699.241839] LustreError: 69175:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff959670333100 x1870603515218432/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1783949096 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2699.271142] LustreError: 69175:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2699.746341] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba441ce730 [ 2704.824954] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2765.808082] Lustre: 69223:0:(ldlm_lib.c:2950:target_recovery_thread()) too long recovery - read logs [ 2765.815500] LustreError: dumping log to /tmp/lustre-log.1783949162.69223 [ 2766.061264] Lustre: lustre-MDT0000: Recovery over after 1:03, of 1 clients 1 recovered and 0 were evicted. [ 2766.072981] Lustre: Skipped 7 previous similar messages [ 2766.128047] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:995 to 0x280000400:1025) [ 2766.129115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:995 to 0x240000400:1025) [ 2773.695452] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2775.380590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2784.539644] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 09:26:19 (1783949179) [ 2793.504171] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2796.095586] Lustre: Failing over lustre-MDT0000 [ 2796.486063] Lustre: server umount lustre-MDT0000 complete [ 2812.899431] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.909519] Lustre: Skipped 15 previous similar messages [ 2818.018139] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2818.030522] LustreError: Skipped 7 previous similar messages [ 2820.069906] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2820.072931] LDISKFS-fs (dm-0): recovery complete [ 2820.078969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.125771] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2837.922988] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2837.935339] Lustre: Skipped 20 previous similar messages [ 2841.952362] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2841.960553] Lustre: Skipped 8 previous similar messages [ 2842.157338] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1027 to 0x280000400:1057) [ 2842.161461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1027 to 0x240000400:1057) [ 2847.800939] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2849.375448] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2859.476653] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 09:27:33 (1783949253) [ 2867.889563] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2870.221799] Lustre: Failing over lustre-MDT0000 [ 2870.802685] Lustre: server umount lustre-MDT0000 complete [ 2894.077121] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2894.080865] LDISKFS-fs (dm-0): recovery complete [ 2894.088355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.027573] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2913.805664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1027 to 0x240000400:1089) [ 2913.807477] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1059 to 0x280000400:1089) [ 2920.220924] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2922.386887] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2931.999593] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 09:28:47 (1783949327) [ 2937.203989] Lustre: 74138:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a7ea1f7e-2cfc-4965-b509-23a966f59454 at adminstrative request [ 2944.614388] Lustre: Failing over lustre-MDT0000 [ 2944.669204] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 2945.054510] Lustre: server umount lustre-MDT0000 complete [ 2965.485241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2966.878635] LustreError: 6977:0:(ofd_io.c:785:ofd_preprw_write()) lustre-OST0000: BRW to missing obj 0x240000400:1090 [ 2973.767716] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2973.779955] Lustre: Skipped 7 previous similar messages [ 2978.984979] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2984.206584] LustreError: 6976:0:(ofd_io.c:785:ofd_preprw_write()) lustre-OST0000: BRW to missing obj 0x240000400:1090 [ 2984.236505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1059 to 0x280000400:1121) [ 2984.236648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1091 to 0x240000400:1121) [ 2990.455585] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2992.171976] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2998.324271] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3009.467947] Lustre: DEBUG MARKER: before 2868, after 2868 [ 3017.578555] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 09:30:12 (1783949412) [ 3018.640371] Lustre: 76186:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a7ea1f7e-2cfc-4965-b509-23a966f59454 at adminstrative request [ 3028.178773] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 09:30:22 (1783949422) [ 3037.912260] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3040.237841] Lustre: Failing over lustre-MDT0000 [ 3040.456930] Lustre: server umount lustre-MDT0000 complete [ 3064.981117] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3064.989371] LDISKFS-fs (dm-0): recovery complete [ 3065.010269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.947200] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95964faa8380 x1870603515305600/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3070.979910] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 3078.233767] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3081.839685] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1124 to 0x280000400:1153) [ 3081.840442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1123 to 0x240000400:1153) [ 3089.960842] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3092.084892] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3101.292679] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 09:31:36 (1783949496) [ 3110.427775] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3112.746250] Lustre: Failing over lustre-MDT0000 [ 3113.047780] Lustre: server umount lustre-MDT0000 complete [ 3138.479563] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3138.482074] LDISKFS-fs (dm-0): recovery complete [ 3138.488974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3143.139655] Lustre: 3449:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783949523/real 1783949523] req@ffff9596761ae300 x1870603515322880/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783949539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3143.222797] Lustre: 3449:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 67 previous similar messages [ 3149.440415] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3158.553814] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1155 to 0x280000400:1185) [ 3158.555628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1155 to 0x240000400:1185) [ 3164.795889] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3166.651514] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3177.331980] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 09:32:52 (1783949572) [ 3186.740370] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3189.213843] Lustre: Failing over lustre-MDT0000 [ 3189.599192] Lustre: server umount lustre-MDT0000 complete [ 3215.545859] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3215.548686] LDISKFS-fs (dm-0): recovery complete [ 3215.556680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.994510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3221.010234] Lustre: Skipped 7 previous similar messages [ 3226.303116] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3235.383495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1187 to 0x240000400:1217) [ 3235.385315] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1187 to 0x280000400:1217) [ 3242.649387] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3244.225753] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3252.818959] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 09:34:07 (1783949647) [ 3261.615495] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3264.337868] Lustre: Failing over lustre-MDT0000 [ 3264.997762] Lustre: server umount lustre-MDT0000 complete [ 3290.193546] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3290.199130] LDISKFS-fs (dm-0): recovery complete [ 3290.223015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.488132] LustreError: 83343:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 3291.501889] LustreError: 83343:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff95967ec0ce00 x1870603515359872/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1783949688 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3291.535570] LustreError: 83343:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 3292.682815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1219 to 0x240000400:1249) [ 3292.691797] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1219 to 0x280000400:1249) [ 3297.657929] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3308.570041] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3310.182170] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3319.765877] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 09:35:14 (1783949714) [ 3328.390534] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3330.329822] Lustre: Failing over lustre-MDT0000 [ 3330.629775] Lustre: server umount lustre-MDT0000 complete [ 3355.237854] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3355.244693] LDISKFS-fs (dm-0): recovery complete [ 3355.271910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.290650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1219 to 0x240000400:1281) [ 3359.295396] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1251 to 0x280000400:1281) [ 3363.919952] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3377.133220] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3379.553050] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3390.257718] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 09:36:24 (1783949784) [ 3400.600391] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3403.710691] Lustre: Failing over lustre-MDT0000 [ 3404.429941] Lustre: server umount lustre-MDT0000 complete [ 3421.619806] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3421.636876] LustreError: Skipped 7 previous similar messages [ 3422.689833] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.711903] Lustre: Skipped 15 previous similar messages [ 3431.221401] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3431.227402] LDISKFS-fs (dm-0): recovery complete [ 3431.244698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.580867] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3446.110960] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3446.127340] Lustre: Skipped 7 previous similar messages [ 3446.319266] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3446.340375] Lustre: Skipped 8 previous similar messages [ 3446.397898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1283 to 0x280000400:1313) [ 3446.399439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1283 to 0x240000400:1313) [ 3460.450308] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3463.840596] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3477.049179] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 09:37:51 (1783949871) [ 3488.747272] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3492.177021] Lustre: Failing over lustre-MDT0000 [ 3492.200534] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 3492.737703] Lustre: server umount lustre-MDT0000 complete [ 3518.467413] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3518.471721] LDISKFS-fs (dm-0): recovery complete [ 3518.490265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3527.164547] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3532.286490] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1315 to 0x280000400:1345) [ 3532.289168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1315 to 0x240000400:1345) [ 3535.335644] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3535.351398] Lustre: Skipped 17 previous similar messages [ 3539.503832] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3541.606559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3551.412509] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 09:39:05 (1783949945) [ 3560.579647] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3562.951832] Lustre: Failing over lustre-MDT0000 [ 3565.325313] Lustre: server umount lustre-MDT0000 complete [ 3590.804704] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3590.812386] LDISKFS-fs (dm-0): recovery complete [ 3590.825540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.127090] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3592.131530] Lustre: Skipped 7 previous similar messages [ 3593.798772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1347 to 0x240000400:1377) [ 3593.798855] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1347 to 0x280000400:1377) [ 3598.338252] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3610.954551] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3613.086816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3623.497782] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 09:40:18 (1783950018) [ 3633.507941] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3636.031558] Lustre: Failing over lustre-MDT0000 [ 3636.360645] Lustre: server umount lustre-MDT0000 complete [ 3663.087654] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3663.089616] LDISKFS-fs (dm-0): recovery complete [ 3663.101541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.503116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1379 to 0x240000400:1409) [ 3666.505127] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1379 to 0x280000400:1409) [ 3670.516748] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3681.730499] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3683.562297] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3695.315479] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 09:41:29 (1783950089) [ 3705.032359] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3708.054390] Lustre: Failing over lustre-MDT0000 [ 3709.416075] LustreError: 93897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3709.441546] LustreError: 93897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3710.483992] Lustre: server umount lustre-MDT0000 complete [ 3738.137953] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3738.142227] LDISKFS-fs (dm-0): recovery complete [ 3738.151306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3750.369801] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95964f096a00 x1870603515479296/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3750.433075] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 12 previous similar messages [ 3751.467479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1411 to 0x240000400:1441) [ 3751.473295] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1411 to 0x280000400:1441) [ 3757.263896] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3768.738856] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3770.937264] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3781.233817] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 09:42:55 (1783950175) [ 3791.704108] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3794.341794] Lustre: Failing over lustre-MDT0000 [ 3794.587162] Lustre: server umount lustre-MDT0000 complete [ 3810.209098] Lustre: 3450:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783950191/real 1783950191] req@ffff95964fc31c00 x1870603515495552/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783950207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3810.217446] Lustre: 3450:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 68 previous similar messages [ 3818.479203] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3818.482267] LDISKFS-fs (dm-0): recovery complete [ 3818.493733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3827.353412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3827.367952] Lustre: Skipped 7 previous similar messages [ 3832.748091] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3835.942423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1443 to 0x240000400:1473) [ 3835.942641] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1443 to 0x280000400:1473) [ 3844.425941] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3846.367581] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3856.544559] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 09:44:11 (1783950251) [ 3857.866373] Lustre: 97890:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a7ea1f7e-2cfc-4965-b509-23a966f59454 at adminstrative request [ 3870.081898] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 09:44:24 (1783950264) [ 3877.979500] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3880.342833] Lustre: Failing over lustre-MDT0000 [ 3881.005265] Lustre: server umount lustre-MDT0000 complete [ 3895.836389] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3895.840423] LDISKFS-fs (dm-0): recovery complete [ 3895.859380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.510648] Lustre: lustre-MDT0000: Aborting client recovery [ 3896.513768] LustreError: 99174:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3896.525310] Lustre: 99220:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3896.535251] Lustre: 99220:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a7ea1f7e-2cfc-4965-b509-23a966f59454@ [ 3896.547691] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3896.560315] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3896.618867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1443 to 0x240000400:1505) [ 3896.621077] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1475 to 0x280000400:1505) [ 3901.701413] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3916.877331] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 09:45:11 (1783950311) [ 3925.909177] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3927.800074] Lustre: Failing over lustre-MDT0000 [ 3927.903521] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 3928.209771] Lustre: server umount lustre-MDT0000 complete [ 3946.399061] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3946.400922] LDISKFS-fs (dm-0): recovery complete [ 3946.410895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3947.392257] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3947.426350] Lustre: lustre-MDT0000: Aborting client recovery [ 3947.428978] LustreError: 100905:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3947.452228] Lustre: 100951:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3947.461600] Lustre: 100951:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 3947.470928] Lustre: 100951:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a7ea1f7e-2cfc-4965-b509-23a966f59454@ [ 3947.480538] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3947.488871] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3947.572633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1511 to 0x240000400:1537) [ 3947.605209] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1511 to 0x280000400:1537) [ 3955.077294] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3961.795133] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3970.092899] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 09:46:04 (1783950364) [ 3979.646820] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3981.836395] Lustre: Failing over lustre-MDT0000 [ 3982.080764] Lustre: server umount lustre-MDT0000 complete [ 3996.656888] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3996.658817] LDISKFS-fs (dm-0): recovery complete [ 3996.667441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.389700] Lustre: lustre-MDT0000: Aborting client recovery [ 3997.402334] LustreError: 102638:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3997.415055] Lustre: 102684:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3997.423470] Lustre: 102684:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 3997.431731] Lustre: 102684:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a7ea1f7e-2cfc-4965-b509-23a966f59454@ [ 3997.438682] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3997.444711] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3997.488685] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1544 to 0x280000400:1569) [ 3997.490696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1543 to 0x240000400:1569) [ 4003.388703] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4015.929691] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 09:46:50 (1783950410) [ 4017.327197] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4017.337771] LustreError: 102652:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95965d4dc700 x1870603497970048/t201863462916(0) o36->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:185/0 lens 512/456 e 0 to 0 dl 1783950425 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 4021.556422] Lustre: Failing over lustre-MDT0000 [ 4021.889393] Lustre: server umount lustre-MDT0000 complete [ 4031.583463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4031.878793] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4031.894811] LustreError: Skipped 8 previous similar messages [ 4032.342158] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4032.351098] Lustre: Skipped 17 previous similar messages [ 4032.464248] Lustre: lustre-MDT0000: Aborting client recovery [ 4032.466955] LustreError: 103827:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4032.471751] Lustre: 103872:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4032.477570] Lustre: 103872:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 4032.482316] Lustre: 103872:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a7ea1f7e-2cfc-4965-b509-23a966f59454@ [ 4032.492655] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 4032.501079] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 4032.559903] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1571 to 0x280000400:1601) [ 4032.577456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1543 to 0x240000400:1601) [ 4037.808956] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4051.463298] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 4053.122424] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 09:47:28 (1783950448) [ 4062.414507] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4065.922761] Lustre: Failing over lustre-MDT0000 [ 4066.411916] Lustre: server umount lustre-MDT0000 complete [ 4081.723870] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4081.730451] LDISKFS-fs (dm-0): recovery complete [ 4081.749639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4082.315453] Lustre: lustre-MDT0000: Aborting client recovery [ 4082.318806] LustreError: 105654:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4082.323552] Lustre: 105701:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4082.329536] Lustre: 105701:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 4082.335610] Lustre: 105701:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a7ea1f7e-2cfc-4965-b509-23a966f59454@ [ 4082.343052] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 4082.348960] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 4082.393200] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1571 to 0x280000400:1633) [ 4082.397500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1543 to 0x240000400:1633) [ 4087.719205] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4102.116767] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 09:48:17 (1783950497) [ 4137.515416] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4139.425118] Lustre: Failing over lustre-MDT0000 [ 4139.934930] Lustre: server umount lustre-MDT0000 complete [ 4162.294045] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4162.295935] LDISKFS-fs (dm-0): recovery complete [ 4162.301420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.262574] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4181.348748] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4181.366192] Lustre: Skipped 5 previous similar messages [ 4181.533185] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4181.543078] Lustre: Skipped 5 previous similar messages [ 4181.604388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2034 to 0x240000400:2049) [ 4181.610547] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2034 to 0x280000400:2049) [ 4184.423171] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4184.431503] Lustre: Skipped 19 previous similar messages [ 4187.131734] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4188.803466] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4210.181249] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 09:50:05 (1783950605) [ 4236.799578] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4246.843727] Lustre: Failing over lustre-MDT0000 [ 4247.162775] Lustre: server umount lustre-MDT0000 complete [ 4271.240220] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4271.242216] LDISKFS-fs (dm-0): recovery complete [ 4271.262497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4276.715247] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba441f9c53 [ 4277.129206] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4277.136218] Lustre: Skipped 19 previous similar messages [ 4281.440664] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4292.097285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2450 to 0x240000400:2465) [ 4292.097770] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2450 to 0x280000400:2465) [ 4298.818620] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4301.167182] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4323.942780] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 09:51:58 (1783950718) [ 4326.389993] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4327.380448] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4327.396418] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4333.757388] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 09:52:08 (1783950728) [ 4364.453219] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4379.814694] Lustre: Failing over lustre-OST0000 [ 4379.921459] Lustre: server umount lustre-OST0000 complete [ 4381.054223] LustreError: 42500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.717241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4383.731661] LustreError: 42510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4386.144698] LustreError: 42505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4388.835521] LustreError: 40453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.963406] LustreError: 42499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.990051] LustreError: 42499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4404.193177] LustreError: 42510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.209403] LustreError: 42510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4404.393605] LDISKFS-fs (dm-1): 3 truncates cleaned up [ 4404.396169] LDISKFS-fs (dm-1): recovery complete [ 4404.409190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4411.953895] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4473.641768] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 09:54:28 (1783950868) [ 4482.387257] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4485.051068] Lustre: Failing over lustre-MDT0000 [ 4485.280986] Lustre: server umount lustre-MDT0000 complete [ 4503.009257] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783950883/real 1783950883] req@ffff9595434c5f80 x1870603516089856/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783950899 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4503.055381] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 50 previous similar messages [ 4509.142265] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4509.147226] LDISKFS-fs (dm-0): recovery complete [ 4509.160000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.224884] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff959647485180 x1870603516092032/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4512.774067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4512.780070] Lustre: Skipped 8 previous similar messages [ 4514.224498] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 4514.226759] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2881) [ 4516.945820] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4527.298637] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4528.705405] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4530.656225] LustreError: 115006:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 4530.657804] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4531.680713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2913) [ 4548.733322] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 09:55:43 (1783950943) [ 4554.410830] LustreError: 114985:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 4559.840402] LustreError: 114985:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4559.856622] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnecting [ 4559.904419] LustreError: 8870:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 waking [ 4561.869495] LustreError: 114986:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 4567.009909] LustreError: 114986:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4567.030962] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnecting [ 4568.883648] LustreError: 114987:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 4574.183140] LustreError: 114987:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4574.211619] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnecting [ 4576.312800] LustreError: 115625:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 4581.346397] LustreError: 115625:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4583.541936] LustreError: 114986:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 4589.024187] LustreError: 114986:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4589.036263] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnecting [ 4589.049833] Lustre: Skipped 1 previous similar message [ 4597.842507] LustreError: 114986:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 4597.846939] LustreError: 114986:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4603.360368] LustreError: 114986:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4603.365532] LustreError: 114986:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4610.529662] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnecting [ 4610.542689] Lustre: Skipped 2 previous similar messages [ 4619.385733] LustreError: 114986:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 4619.393912] LustreError: 114986:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4624.864458] LustreError: 114986:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4624.878435] LustreError: 114986:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4633.977404] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 09:57:08 (1783951028) [ 4635.732203] LustreError: 116585:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4646.237828] Lustre: lustre-MDT0000: Export ffff95964371e000 already connecting from 192.168.202.13@tcp [ 4648.285846] Lustre: lustre-MDT0000: Export ffff95964371e000 already connecting from 192.168.202.13@tcp [ 4651.294169] Lustre: lustre-MDT0000: Export ffff95964371e000 already connecting from 192.168.202.13@tcp [ 4656.483085] Lustre: lustre-MDT0000: Export ffff95964371e000 already connecting from 192.168.202.13@tcp [ 4661.614081] Lustre: lustre-MDT0000: Export ffff95964371e000 already connecting from 192.168.202.13@tcp [ 4661.622792] Lustre: Skipped 1 previous similar message [ 4671.838927] Lustre: lustre-MDT0000: Export ffff95964371e000 already connecting from 192.168.202.13@tcp [ 4671.851885] Lustre: Skipped 1 previous similar message [ 4675.810440] LustreError: 116585:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4675.821553] Lustre: 116585:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9596448d6a00 x1870603500616704/t0(0) o38->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:0/0 lens 520/416 e 0 to 0 dl 1783951052 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 4676.958264] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnecting [ 4676.974280] Lustre: Skipped 3 previous similar messages [ 4676.977049] LustreError: 114987:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4702.562238] Lustre: lustre-MDT0000: Export ffff95964371e000 already connecting from 192.168.202.13@tcp [ 4717.000462] LustreError: 114987:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4717.005416] Lustre: 114987:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9596675a3480 x1870603500620032/t0(0) o38->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:0/0 lens 520/416 e 0 to 0 dl 1783951093 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4717.921363] LustreError: 115625:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4743.516106] Lustre: lustre-MDT0000: Export ffff95964371e000 already connecting from 192.168.202.13@tcp [ 4743.518825] Lustre: Skipped 3 previous similar messages [ 4757.968155] LustreError: 115625:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4757.982236] Lustre: 115625:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff95967f315500 x1870603500621952/t0(0) o38->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:0/0 lens 520/416 e 0 to 0 dl 1783951134 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4758.880679] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnecting [ 4758.887179] Lustre: Skipped 1 previous similar message [ 4758.892056] LustreError: 114987:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4798.905592] LustreError: 114987:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4798.916408] Lustre: 114987:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff959543161c00 x1870603500623872/t0(0) o38->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:0/0 lens 520/416 e 0 to 0 dl 1783951175 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4799.843571] LustreError: 116585:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4825.446478] Lustre: lustre-MDT0000: Export ffff95964371e000 already connecting from 192.168.202.13@tcp [ 4825.460386] Lustre: Skipped 7 previous similar messages [ 4839.852043] LustreError: 116585:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4839.875942] Lustre: 116585:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9596763c8700 x1870603500625792/t0(0) o38->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:0/0 lens 520/416 e 0 to 0 dl 1783951216 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4840.476470] LustreError: 115625:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4863.352074] LustreError: 115625:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout interrupted [ 4863.355895] Lustre: 115625:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/3s); client may timeout req@ffff95964f16aa00 x1870603500627456/t0(0) o38->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:0/0 lens 520/416 e 0 to 0 dl 1783951257 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 4869.029699] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 10:01:03 (1783951263) [ 4877.340284] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4881.854718] Lustre: Failing over lustre-MDT0000 [ 4882.119766] Lustre: server umount lustre-MDT0000 complete [ 4895.131359] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4895.134468] LDISKFS-fs (dm-0): recovery complete [ 4895.139832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4895.259557] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4895.268370] LustreError: Skipped 4 previous similar messages [ 4895.527193] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.529076] Lustre: *** cfs_fail_loc=712, val=0*** [ 4895.547936] Lustre: Skipped 12 previous similar messages [ 4895.565975] LustreError: 6972:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff95964f42ed80 x1870603516197248/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 4895.698775] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4895.699155] Lustre: lustre-MDT0000: Aborting client recovery [ 4895.708656] Lustre: Skipped 2 previous similar messages [ 4895.722756] LustreError: 119874:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4895.730391] Lustre: 119920:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4895.733139] Lustre: 119920:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 4895.745398] Lustre: 119920:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a7ea1f7e-2cfc-4965-b509-23a966f59454@ [ 4895.751329] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 4895.761767] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 4895.829933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2913) [ 4895.835556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2945) [ 4900.389435] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4900.837531] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4900.848179] Lustre: Skipped 9 previous similar messages [ 4908.666853] Lustre: Failing over lustre-MDT0000 [ 4909.303903] Lustre: server umount lustre-MDT0000 complete [ 4929.657892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4936.354690] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba442186d8 [ 4938.095045] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4938.110533] Lustre: Skipped 3 previous similar messages [ 4938.172160] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4938.176416] Lustre: Skipped 3 previous similar messages [ 4938.241798] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2945) [ 4938.247733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2977) [ 4942.055563] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4952.076845] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4953.976537] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4962.598620] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 10:02:37 (1783951357) [ 4962.692124] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnecting [ 4962.708812] Lustre: Skipped 2 previous similar messages [ 4970.247514] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 10:02:45 (1783951365) [ 4971.363833] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4971.370641] LustreError: 120816:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95964f42c380 x1870603500683904/t0(0) o700->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:384/0 lens 264/248 e 0 to 0 dl 1783951379 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 4990.302660] Lustre: Failing over lustre-MDT0000 [ 4990.681784] Lustre: server umount lustre-MDT0000 complete [ 5012.591955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5024.192135] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5032.890326] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2947 to 0x280000400:2977) [ 5032.896034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2979 to 0x240000400:3009) [ 5039.064599] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5040.258215] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5050.967506] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 10:04:05 (1783951445) [ 5054.134780] Lustre: Failing over lustre-OST0000 [ 5054.373822] Lustre: server umount lustre-OST0000 complete [ 5058.432729] LustreError: 42510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5058.529299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5063.529749] LustreError: 6972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5063.556691] LustreError: 6972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5068.650495] LustreError: 42500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5068.691837] LustreError: 42500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5074.981636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5081.715451] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5091.355882] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5093.451820] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5170.823108] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 10:06:05 (1783951565) [ 5180.677674] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5183.676104] Lustre: Failing over lustre-MDT0000 [ 5184.162150] Lustre: server umount lustre-MDT0000 complete [ 5203.424119] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783951584/real 1783951584] req@ffff95965d4dc380 x1870603516284416/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783951600 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5203.440349] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 33 previous similar messages [ 5209.474120] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5209.477598] LDISKFS-fs (dm-0): recovery complete [ 5209.485090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.674283] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba44219dbb [ 5214.014897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5214.025338] Lustre: Skipped 4 previous similar messages [ 5218.943273] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5227.857732] Lustre: *** cfs_fail_loc=216, val=0*** [ 5227.858692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3030 to 0x240000400:3073) [ 5227.866504] LustreError: 125938:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 5228.962497] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2998 to 0x280000400:3041) [ 5296.398650] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 10:08:11 (1783951691) [ 5298.387686] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5298.392304] Lustre: Skipped 2 previous similar messages [ 5313.649436] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 10:08:28 (1783951708) [ 5317.129929] Lustre: Failing over lustre-MDT0000 [ 5317.407673] Lustre: server umount lustre-MDT0000 complete [ 5336.850848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5337.051784] LustreError: 127497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5337.086477] LustreError: 127497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5341.742438] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5345.132666] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5345.141428] LustreError: 127531:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95965d478a80 x1870603500806784/t0(0) o101->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:2/0 lens 328/344 e 0 to 0 dl 1783951752 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 5361.500052] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnected, waiting for 1 clients in recovery for 1:24 [ 5361.699815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3085 to 0x240000400:3105) [ 5361.701644] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3052 to 0x280000400:3073) [ 5369.762113] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5372.038653] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5383.737900] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 10:09:38 (1783951778) [ 5385.808951] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5394.988395] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5397.365562] Lustre: Failing over lustre-MDT0000 [ 5398.053829] Lustre: server umount lustre-MDT0000 complete [ 5423.127626] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5423.131506] LDISKFS-fs (dm-0): recovery complete [ 5423.141632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5424.288103] LustreError: 129603:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 5424.300245] LustreError: 129603:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff95967f315c00 x1870603516343168/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1783951821 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5424.321637] LustreError: 129603:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 5424.608720] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95965d478a80 x1870603516343552/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5424.630670] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 100 previous similar messages [ 5427.638173] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3075 to 0x280000400:3105) [ 5427.638284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3085 to 0x240000400:3137) [ 5429.768049] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5440.633852] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5443.104070] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5452.947587] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 10:10:47 (1783951847) [ 5454.423885] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5465.078409] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5467.740545] Lustre: Failing over lustre-MDT0000 [ 5468.134654] Lustre: server umount lustre-MDT0000 complete [ 5491.594411] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5491.596914] LDISKFS-fs (dm-0): recovery complete [ 5491.603255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.652960] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba4421bc77 [ 5495.964813] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5495.973259] Lustre: Skipped 8 previous similar messages [ 5496.263790] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3075 to 0x280000400:3137) [ 5496.278480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3169) [ 5500.579890] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5510.213335] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5510.224099] Lustre: Skipped 18 previous similar messages [ 5511.950599] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5514.088582] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5524.591326] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 10:11:59 (1783951919) [ 5525.583963] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5527.609496] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5537.996461] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5540.119393] Lustre: Failing over lustre-MDT0000 [ 5540.570139] Lustre: server umount lustre-MDT0000 complete [ 5556.195215] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5556.204659] Lustre: Skipped 16 previous similar messages [ 5562.336361] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5562.354357] LustreError: Skipped 6 previous similar messages [ 5564.767785] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5564.769891] LDISKFS-fs (dm-0): recovery complete [ 5564.776379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5577.791729] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5586.717850] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5586.731386] Lustre: Skipped 6 previous similar messages [ 5586.765215] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5586.768572] Lustre: Skipped 6 previous similar messages [ 5586.865514] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3075 to 0x280000400:3169) [ 5586.867808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3171 to 0x240000400:3201) [ 5597.591417] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 10:13:12 (1783951992) [ 5600.073024] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5600.077767] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 5600.086025] LustreError: 133711:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95964f169880 x1870603500862464/t257698037777(0) o35->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:257/0 lens 392/456 e 0 to 0 dl 1783952007 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5603.647489] Lustre: Failing over lustre-MDT0000 [ 5604.192238] Lustre: server umount lustre-MDT0000 complete [ 5624.336245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5638.310109] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5641.142070] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3075 to 0x280000400:3201) [ 5641.143701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3203 to 0x240000400:3233) [ 5641.152259] Lustre: 134972:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95967a002680 x1870603500862464/t257698037777(0) o35->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:298/0 lens 392/456 e 0 to 0 dl 1783952048 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5649.393952] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5651.485422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5660.656206] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 10:14:15 (1783952055) [ 5662.036700] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5662.042790] LustreError: 134969:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9596675a1f80 x1870603500876672/t261993005072(0) o36->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:319/0 lens 504/448 e 0 to 0 dl 1783952069 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5672.066576] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5674.449577] Lustre: Failing over lustre-MDT0000 [ 5674.787248] Lustre: server umount lustre-MDT0000 complete [ 5701.057570] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5701.060409] LDISKFS-fs (dm-0): recovery complete [ 5701.068203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5711.841058] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5723.547683] Lustre: 137019:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95967a160a80 x1870603500876672/t261993005072(0) o36->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:381/0 lens 504/2880 e 0 to 0 dl 1783952131 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5723.559918] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3233) [ 5723.560444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3203 to 0x240000400:3265) [ 5731.663554] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5733.211063] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5742.581850] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 10:15:37 (1783952137) [ 5743.865918] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5743.877347] LustreError: 137017:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9596448d4380 x1870603500891776/t266287972368(0) o36->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:401/0 lens 504/448 e 0 to 0 dl 1783952151 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5746.073024] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5754.420467] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5756.796806] Lustre: Failing over lustre-MDT0000 [ 5757.228424] Lustre: server umount lustre-MDT0000 complete [ 5782.673772] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5782.680382] LDISKFS-fs (dm-0): recovery complete [ 5782.690626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.516246] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5805.498236] Lustre: 139089:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95964f42f480 x1870603500892416/t266287972369(0) o35->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:463/0 lens 392/456 e 0 to 0 dl 1783952213 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5805.513622] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3265) [ 5805.515495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3267 to 0x240000400:3297) [ 5805.536200] Lustre: 139089:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5818.647607] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 10:16:52 (1783952212) [ 5819.962661] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5819.967580] Lustre: Skipped 1 previous similar message [ 5819.972411] LustreError: 139483:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9596764a9180 x1870603500905856/t270582939664(0) o36->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:477/0 lens 504/448 e 0 to 0 dl 1783952227 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5820.014060] LustreError: 139483:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 1 previous similar message [ 5822.097982] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5832.045816] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5834.269587] Lustre: Failing over lustre-MDT0000 [ 5834.641314] Lustre: server umount lustre-MDT0000 complete [ 5853.537628] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952234/real 1783952234] req@ffff95967f315c00 x1870603516453376/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783952250 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5853.568791] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 61 previous similar messages [ 5858.672449] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5858.675342] LDISKFS-fs (dm-0): recovery complete [ 5858.691231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.161452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5865.165766] Lustre: Skipped 7 previous similar messages [ 5870.008919] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5881.249871] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3297) [ 5881.251125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3299 to 0x240000400:3329) [ 5881.277453] Lustre: 141011:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff959645a5aa00 x1870603500905856/t270582939664(0) o36->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:539/0 lens 504/2880 e 0 to 0 dl 1783952289 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5891.323671] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 10:18:06 (1783952286) [ 5892.813829] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5894.648524] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5894.650481] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 5894.661387] LustreError: 141012:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95967d87c000 x1870603500919040/t274877906960(0) o35->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:552/0 lens 392/456 e 0 to 0 dl 1783952302 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5903.969150] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5906.143758] Lustre: Failing over lustre-MDT0000 [ 5906.443265] Lustre: server umount lustre-MDT0000 complete [ 5929.841653] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5929.849224] LDISKFS-fs (dm-0): recovery complete [ 5929.877255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.358445] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5955.530990] Lustre: 142834:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95967f316300 x1870603500919040/t274877906960(0) o35->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:613/0 lens 392/456 e 0 to 0 dl 1783952363 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5955.549675] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3329) [ 5955.550033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3331 to 0x240000400:3361) [ 5965.416680] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 10:19:20 (1783952360) [ 5966.723962] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 5966.734405] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5966.738223] LustreError: 142833:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9596675a1880 x1870603500930560/t279172874255(0) o101->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:624/0 lens 664/608 e 0 to 0 dl 1783952374 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5983.072044] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnecting [ 5983.078850] Lustre: Skipped 1 previous similar message [ 5983.092325] Lustre: 142833:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95967f316300 x1870603500930560/t279172874255(0) o101->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:640/0 lens 664/3488 e 0 to 0 dl 1783952390 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5994.534700] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 10:19:48 (1783952388) [ 6005.034363] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6007.254200] Lustre: Failing over lustre-MDT0000 [ 6007.293271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6007.758930] Lustre: server umount lustre-MDT0000 complete [ 6033.141775] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 6033.145641] LDISKFS-fs (dm-0): recovery complete [ 6033.155948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6043.423649] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6052.782242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3331 to 0x240000400:3393) [ 6052.783080] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3331 to 0x280000400:3361) [ 6057.044681] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6058.942478] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6078.643574] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 10:21:13 (1783952473) [ 6088.418914] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6090.836782] Lustre: Failing over lustre-MDT0000 [ 6091.282435] Lustre: server umount lustre-MDT0000 complete [ 6115.376493] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6115.379653] LDISKFS-fs (dm-0): recovery complete [ 6115.397159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6120.225583] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95964f169c00 x1870603516514304/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6120.246532] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 6120.834408] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6120.858514] Lustre: Skipped 7 previous similar messages [ 6127.855485] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6134.802943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3395 to 0x240000400:3425) [ 6134.803334] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3331 to 0x280000400:3393) [ 6135.014815] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6135.027460] Lustre: Skipped 14 previous similar messages [ 6141.571923] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6143.259211] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6149.480481] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 6162.668652] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 10:22:37 (1783952557) [ 6212.071107] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6214.102248] Lustre: Failing over lustre-MDT0000 [ 6214.607860] Lustre: server umount lustre-MDT0000 complete [ 6232.544251] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.545341] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6232.559352] Lustre: Skipped 16 previous similar messages [ 6232.577706] LustreError: Skipped 7 previous similar messages [ 6238.669874] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6238.673371] LDISKFS-fs (dm-0): recovery complete [ 6238.690984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6248.657695] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6257.512504] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 6257.522237] Lustre: Skipped 7 previous similar messages [ 6257.875896] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 6257.879821] Lustre: Skipped 7 previous similar messages [ 6257.901848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4676 to 0x240000400:4705) [ 6257.902993] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4644 to 0x280000400:4673) [ 6264.096899] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6266.192830] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6334.765429] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 10:25:29 (1783952729) [ 6346.470594] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6348.268101] Lustre: Failing over lustre-MDT0000 [ 6348.613889] Lustre: server umount lustre-MDT0000 complete [ 6372.933062] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6372.935660] LDISKFS-fs (dm-0): recovery complete [ 6372.944657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6375.402157] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba4425f77f [ 6378.390723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4707 to 0x240000400:4737) [ 6378.391838] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4644 to 0x280000400:4705) [ 6379.698792] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6390.598662] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6392.197339] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6403.923899] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 6406.015628] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 6416.000985] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 10:26:50 (1783952810) [ 6424.927315] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 6444.282093] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6444.286041] LustreError: 151703:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff959642e50450 x1870603503716608/t296352743435(0) o36->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:347/0 lens 66040/440 e 0 to 0 dl 1783952852 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 6459.764496] Lustre: 151703:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95965d4d5c00 x1870603503716608/t296352743435(0) o36->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:362/0 lens 66040/440 e 0 to 0 dl 1783952867 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 6471.187335] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 6473.004983] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 10:27:47 (1783952867) [ 6488.716456] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6491.593435] Lustre: Failing over lustre-MDT0000 [ 6491.981474] Lustre: server umount lustre-MDT0000 complete [ 6509.024344] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952889/real 1783952889] req@ffff95966f667480 x1870603517255296/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783952905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6509.051674] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 48 previous similar messages [ 6514.526722] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6514.530447] LDISKFS-fs (dm-0): recovery complete [ 6514.545815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6518.264265] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba442625c3 [ 6518.781597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6518.792164] Lustre: Skipped 5 previous similar messages [ 6523.781031] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6524.273640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4838 to 0x240000400:4865) [ 6524.283670] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4807 to 0x280000400:4833) [ 6537.355149] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6539.337497] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6550.442163] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 10:29:05 (1783952945) [ 6574.372374] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6589.979736] Lustre: Failing over lustre-OST0000 [ 6590.081751] Lustre: server umount lustre-OST0000 complete [ 6590.433776] LustreError: 42503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6590.447570] LustreError: 42503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6595.561601] LustreError: 42508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6600.673700] LustreError: 6969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6610.926863] LustreError: 150282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6610.959909] LustreError: 150282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6615.698692] LDISKFS-fs (dm-1): 3 truncates cleaned up [ 6615.708391] LDISKFS-fs (dm-1): recovery complete [ 6615.741664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6622.988656] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6637.713893] Lustre: Failing over lustre-OST0000 [ 6637.848685] Lustre: server umount lustre-OST0000 complete [ 6639.587703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6639.596113] LustreError: 6971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6655.492292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6662.430069] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6671.147664] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6672.821385] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6712.319658] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 10:31:47 (1783953107) [ 6715.210607] Lustre: Failing over lustre-MDT0000 [ 6715.568331] Lustre: server umount lustre-MDT0000 complete [ 6733.230137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6733.643867] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6733.649787] Lustre: Skipped 5 previous similar messages [ 6738.072690] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6738.919107] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6738.926716] Lustre: Skipped 11 previous similar messages [ 6743.492423] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5234 to 0x280000400:5249) [ 6743.501521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5266 to 0x240000400:5281) [ 6753.175702] Lustre: Failing over lustre-MDT0000 [ 6753.353910] Lustre: server umount lustre-MDT0000 complete [ 6774.062523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6786.073405] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6794.659044] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5234 to 0x280000400:5281) [ 6794.660920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5266 to 0x240000400:5313) [ 6801.900556] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6803.859113] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6813.838443] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 10:33:28 (1783953208) [ 6828.021040] Lustre: Failing over lustre-OST0000 [ 6828.269774] Lustre: server umount lustre-OST0000 complete [ 6830.445734] LustreError: 150282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6830.467568] LustreError: 150282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6830.566643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6848.458531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6856.908846] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6867.371610] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6869.028499] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6879.065794] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 10:34:34 (1783953274) [ 6881.094846] Lustre: Failing over lustre-MDT0000 [ 6881.272760] Lustre: server umount lustre-MDT0000 complete [ 6891.530710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6891.683672] Lustre: *** cfs_fail_loc=605, val=0*** [ 6891.685388] LustreError: 163307:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0996100 failed: rc = -95 [ 6891.704448] LustreError: 163307:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 6891.711873] LustreError: 163307:0:(obd_mount.c:259:lustre_start_simple()) MGS setup error -95 [ 6891.727904] LustreError: 163307:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 6891.737335] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 6891.746743] LustreError: 163307:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6891.772200] Lustre: server umount lustre-MDT0000 complete [ 6891.779140] LustreError: 163307:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 6900.196891] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6900.205505] LustreError: Skipped 4 previous similar messages [ 6901.285243] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6901.295825] Lustre: Skipped 11 previous similar messages [ 6901.475357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.432555] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95965d4dc380 x1870603517488640/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6910.457839] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 14 previous similar messages [ 6915.326359] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6922.601549] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 6922.608234] Lustre: Skipped 7 previous similar messages [ 6922.627073] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 6922.643170] Lustre: Skipped 7 previous similar messages [ 6922.693056] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5283 to 0x280000400:5313) [ 6922.697404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5315 to 0x240000400:5345) [ 6925.582683] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 10:35:20 (1783953320) [ 6934.446833] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6938.083704] Lustre: Failing over lustre-MDT0000 [ 6938.511562] Lustre: server umount lustre-MDT0000 complete [ 6963.316329] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6963.318867] LDISKFS-fs (dm-0): recovery complete [ 6963.327802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.144171] Lustre: *** cfs_fail_loc=707, val=0*** [ 6973.445170] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6985.599323] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnected, waiting for 1 clients in recovery for 0:54 [ 6986.373313] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5326 to 0x280000400:5345) [ 6986.373396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5359 to 0x240000400:5377) [ 6994.299985] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6996.298027] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7006.859449] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 10:36:41 (1783953401) [ 7039.084127] LustreError: 165473:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff959542ff4e00 x1870603504594176/t0(0) o101->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:186/0 lens 664/0 e 0 to 0 dl 1783953446 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7039.117438] LustreError: 165473:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 7050.168416] LustreError: 165473:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7050.182338] LustreError: 165476:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff959542ff5f80 x1870603504595200/t0(0) o35->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:203/0 lens 392/0 e 0 to 0 dl 1783953463 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7052.926400] LustreError: 165472:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff959542ff6d80 x1870603504601088/t0(0) o101->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:239/0 lens 576/0 e 0 to 0 dl 1783953499 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 7052.958494] LustreError: 165472:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 27 previous similar messages [ 7055.845316] LustreError: 42498:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff959542710000 x1870603517530880/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:203/0 lens 544/0 e 0 to 0 dl 1783953463 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 7055.873305] LustreError: 42498:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 20 previous similar messages [ 7063.537514] LustreError: 42505:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff959541c0a680 x1870603517532288/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:211/0 lens 544/0 e 0 to 0 dl 1783953471 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 7063.563203] LustreError: 42505:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 7 previous similar messages [ 7069.151893] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 10:37:44 (1783953464) [ 7099.143423] LustreError: 41250:0:(tgt_handler.c:2836:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 7110.208164] LustreError: 41250:0:(tgt_handler.c:2836:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 7120.278584] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 10:38:35 (1783953515) [ 7150.050243] LustreError: 165473:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff959542652300 x1870603504619136/t0(0) o101->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:297/0 lens 576/0 e 0 to 0 dl 1783953557 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7150.076104] LustreError: 165473:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 7155.080144] LustreError: 165473:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7158.556625] LustreError: 166184:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 10000ms [ 7168.594474] LustreError: 166184:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7168.610129] LustreError: 165474:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff95964f16ad80 x1870603504641792/t0(0) o101->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:355/0 lens 664/0 e 0 to 0 dl 1783953615 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7168.634283] LustreError: 165474:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 120 previous similar messages [ 7187.410272] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 10:39:42 (1783953582) [ 7281.833875] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 10:41:16 (1783953676) [ 7311.309410] LustreError: 165472:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff95967a15f480 x1870603504697984/t0(0) o101->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:459/0 lens 576/0 e 0 to 0 dl 1783953719 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7311.350402] LustreError: 165472:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 7311.360179] LustreError: 165472:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 7311.785819] LustreError: 165472:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7328.146437] LustreError: 165879:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7328.154585] LustreError: 165879:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 7343.672559] LustreError: 165473:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 7343.692041] LustreError: 165473:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 77 previous similar messages [ 7361.352435] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 10:42:36 (1783953756) [ 7394.172098] Lustre: DEBUG MARKER: phase 2 [ 7402.954505] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 10:43:17 (1783953797) [ 7485.466423] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 10:44:40 (1783953880) [ 7486.951859] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 7488.709966] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 10:44:43 (1783953883) [ 7493.409573] Lustre: DEBUG MARKER: Started rundbench load pid=132847 ... [ 7503.182601] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7505.802073] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 7507.927798] Lustre: Failing over lustre-MDT0000 [ 7507.982308] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 7507.992452] Lustre: Skipped 1 previous similar message [ 7508.348744] Lustre: server umount lustre-MDT0000 complete [ 7528.929530] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783953909/real 1783953909] req@ffff9596764ae300 x1870603517665920/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783953925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7528.931798] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7528.953603] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 44 previous similar messages [ 7528.953753] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7528.953757] LustreError: Skipped 1 previous similar message [ 7529.028902] Lustre: Skipped 4 previous similar messages [ 7531.322263] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7531.328788] LDISKFS-fs (dm-0): recovery complete [ 7531.338329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7539.169536] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff959542677100 x1870603517667968/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7539.579574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7539.587690] Lustre: Skipped 7 previous similar messages [ 7539.633628] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7539.645422] Lustre: Skipped 4 previous similar messages [ 7543.818356] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7544.361454] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 7544.370284] Lustre: Skipped 1 previous similar message [ 7545.652645] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 7545.659177] Lustre: Skipped 1 previous similar message [ 7545.720319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5481 to 0x240000400:5505) [ 7545.724537] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5428 to 0x280000400:5473) [ 7552.459085] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7554.074415] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7555.050618] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7555.063771] Lustre: Skipped 8 previous similar messages [ 7564.593794] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7567.375130] Lustre: DEBUG MARKER: test_70b fail mds1 2 times [ 7569.429633] Lustre: Failing over lustre-MDT0000 [ 7569.443141] LustreError: 3451:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff959542677480 x1870603517750400/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 7569.459046] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 7569.781338] Lustre: server umount lustre-MDT0000 complete [ 7592.350772] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7592.354994] LDISKFS-fs (dm-0): recovery complete [ 7592.366245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7597.031614] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba4428270f [ 7602.158342] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7606.062864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5543 to 0x240000400:5569) [ 7606.062954] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5511 to 0x280000400:5537) [ 7611.970678] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7613.648169] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7644.963377] Lustre: DEBUG MARKER: == replay-single test 70c: tar 1mdts recovery ============ 10:47:19 (1783954039) [ 7773.309995] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7786.166086] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 7788.808430] Lustre: Failing over lustre-MDT0000 [ 7788.859800] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 7789.481946] Lustre: server umount lustre-MDT0000 complete [ 7816.461531] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7816.469791] LDISKFS-fs (dm-0): recovery complete [ 7816.512235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.681837] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba442d2346 [ 7826.809658] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7837.107955] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6303 to 0x280000400:6337) [ 7837.110348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6336 to 0x240000400:6369) [ 7843.797880] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7845.811480] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7930.501678] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 10:52:04 (1783954324) [ 7932.840992] Lustre: DEBUG MARKER: SKIP: replay-single test_70d needs >= 2 MDTs [ 7935.500117] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 10:52:09 (1783954329) [ 7938.514132] Lustre: DEBUG MARKER: SKIP: replay-single test_70e needs >= 2 MDTs [ 7940.637573] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 10:52:15 (1783954335) [ 7955.225295] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7957.930175] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7959.738200] Lustre: Failing over lustre-OST0000 [ 7959.987527] Lustre: server umount lustre-OST0000 complete [ 7960.068687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7960.080590] LustreError: 15370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7960.120947] LustreError: 15370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7971.166159] LustreError: 42504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.189018] LustreError: 42504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7985.712309] LDISKFS-fs (dm-1): 3 truncates cleaned up [ 7985.718095] LDISKFS-fs (dm-1): recovery complete [ 7985.732645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7993.625217] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8005.577833] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8008.172797] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8019.748848] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 10:53:35 (1783954415) [ 8021.553593] Lustre: DEBUG MARKER: SKIP: replay-single test_71a needs >= 2 MDTs [ 8023.428749] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 10:53:38 (1783954418) [ 8032.345652] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8035.252881] Lustre: Failing over lustre-MDT0000 [ 8035.685859] Lustre: server umount lustre-MDT0000 complete [ 8060.166680] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8060.173290] LDISKFS-fs (dm-0): recovery complete [ 8060.183052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8063.983786] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba44302482 [ 8065.988039] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 8069.388265] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8082.325190] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnected, waiting for 1 clients in recovery for 0:53 [ 8082.406858] Lustre: 181477:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95965d5dd500 x1870603513143936/t330712486984(330712486984) o101->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:475/0 lens 592/3488 e 0 to 0 dl 1783954490 ref 1 fl Interpret:/606/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 8082.599410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6591 to 0x240000400:6625) [ 8082.600255] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6558 to 0x280000400:6593) [ 8089.122270] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8091.098321] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8101.746357] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 10:54:56 (1783954496) [ 8109.603223] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8112.565941] Lustre: Failing over lustre-MDT0000 [ 8114.657948] LustreError: 181478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.681067] LustreError: 181478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8114.959250] Lustre: server umount lustre-MDT0000 complete [ 8130.854264] Lustre: 3450:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783954511/real 1783954511] req@ffff959667631500 x1870603518721024/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783954527 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8130.900856] Lustre: 3450:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [ 8130.911069] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8130.923956] LustreError: Skipped 3 previous similar messages [ 8138.340424] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8138.344419] LDISKFS-fs (dm-0): recovery complete [ 8138.349434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.192521] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff959643757480 x1870603518722688/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8140.235269] LustreError: 3448:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 21 previous similar messages [ 8140.263092] Lustre: MGS: Client 95bdee25-7093-409e-b50e-2b2d9c2bc297 (at 0@lo) reconnecting [ 8140.274773] Lustre: Skipped 2 previous similar messages [ 8140.672640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8140.678696] Lustre: Skipped 4 previous similar messages [ 8140.704253] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8140.707133] Lustre: Skipped 4 previous similar messages [ 8144.321921] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 8144.337225] LustreError: 183495:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff959643756a00 x1870603513156224/t335007449091(335007449091) o101->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:537/0 lens 592/608 e 0 to 0 dl 1783954552 ref 1 fl Interpret:/604/0 rc 301/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 8145.469069] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8155.107242] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 8155.113638] Lustre: Skipped 12 previous similar messages [ 8159.647728] Lustre: lustre-MDT0000: Client a7ea1f7e-2cfc-4965-b509-23a966f59454 (at 192.168.202.13@tcp) reconnected, waiting for 1 clients in recovery for 0:54 [ 8159.680153] Lustre: 183494:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9596458fce00 x1870603513156224/t335007449091(335007449091) o101->a7ea1f7e-2cfc-4965-b509-23a966f59454@192.168.202.13@tcp:552/0 lens 592/3488 e 0 to 0 dl 1783954567 ref 1 fl Interpret:/606/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 8159.750234] Lustre: lustre-MDT0000: Recovery over after 0:15, of 1 clients 1 recovered and 0 were evicted. [ 8159.756881] Lustre: Skipped 4 previous similar messages [ 8159.808521] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6595 to 0x280000400:6625) [ 8159.811627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6591 to 0x240000400:6657) [ 8167.251740] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8169.189550] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8180.489133] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 10:56:15 (1783954575) [ 8185.567526] Lustre: Failing over lustre-OST0000 [ 8185.769575] Lustre: server umount lustre-OST0000 complete [ 8185.826939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8185.852253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8185.869847] Lustre: Skipped 9 previous similar messages [ 8185.876549] LustreError: 42504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8185.890378] LustreError: 42504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8189.946765] Lustre: Failing over lustre-MDT0000 [ 8190.436348] Lustre: server umount lustre-MDT0000 complete [ 8209.784697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.551670] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba44302d9d [ 8216.859231] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6595 to 0x280000400:6657) [ 8222.429216] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8232.158102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8234.235960] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8234.239396] Lustre: Skipped 5 previous similar messages [ 8234.241583] Lustre: lustre-OST0000: Denying connection for new client 6279c5be-5e49-456e-91d8-0d6531caccda (at 192.168.202.13@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 8234.263927] Lustre: Skipped 11 previous similar messages [ 8237.595170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6591 to 0x240000400:6689) [ 8239.667716] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8250.916497] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 10:57:25 (1783954645) [ 8252.724932] Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs [ 8254.449381] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 10:57:29 (1783954649) [ 8256.111414] Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs [ 8258.109173] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 10:57:32 (1783954652) [ 8260.120469] Lustre: DEBUG MARKER: SKIP: replay-single test_80c needs >= 2 MDTs [ 8262.272349] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 10:57:37 (1783954657) [ 8263.974150] Lustre: DEBUG MARKER: SKIP: replay-single test_80d needs >= 2 MDTs [ 8266.143514] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 10:57:40 (1783954660) [ 8268.033352] Lustre: DEBUG MARKER: SKIP: replay-single test_80e needs >= 2 MDTs [ 8270.417387] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 10:57:44 (1783954664) [ 8272.578713] Lustre: DEBUG MARKER: SKIP: replay-single test_80f needs >= 2 MDTs [ 8274.661799] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 10:57:49 (1783954669) [ 8276.958334] Lustre: DEBUG MARKER: SKIP: replay-single test_80g needs >= 2 MDTs [ 8279.459569] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 10:57:53 (1783954673) [ 8281.858895] Lustre: DEBUG MARKER: SKIP: replay-single test_80h needs >= 2 MDTs [ 8284.151478] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 10:57:58 (1783954678) [ 8286.461860] Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs [ 8288.600342] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 10:58:03 (1783954683) [ 8290.106795] Lustre: DEBUG MARKER: SKIP: replay-single test_81b needs >= 2 MDTs [ 8292.203279] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 10:58:06 (1783954686) [ 8293.942762] Lustre: DEBUG MARKER: SKIP: replay-single test_81c needs >= 2 MDTs [ 8295.545418] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 10:58:10 (1783954690) [ 8297.083869] Lustre: DEBUG MARKER: SKIP: replay-single test_81d needs >= 2 MDTs [ 8298.922347] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 10:58:13 (1783954693) [ 8301.046497] Lustre: DEBUG MARKER: SKIP: replay-single test_81e needs >= 2 MDTs [ 8302.994695] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 10:58:18 (1783954698) [ 8304.859605] Lustre: DEBUG MARKER: SKIP: replay-single test_81f needs >= 2 MDTs [ 8307.304227] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 10:58:21 (1783954701) [ 8309.671565] Lustre: DEBUG MARKER: SKIP: replay-single test_81g needs >= 2 MDTs [ 8312.745295] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 10:58:26 (1783954706) [ 8314.957981] Lustre: DEBUG MARKER: SKIP: replay-single test_81h needs >= 2 MDTs [ 8317.045132] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 10:58:31 (1783954711) [ 8319.009537] Lustre: 188048:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 6279c5be-5e49-456e-91d8-0d6531caccda at adminstrative request [ 8330.867548] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 10:58:45 (1783954725) [ 8340.192184] Lustre: Failing over lustre-MDT0000 [ 8340.435055] Lustre: server umount lustre-MDT0000 complete [ 8359.421654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.759495] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8369.850711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6741 to 0x240000400:6785) [ 8369.850923] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6709 to 0x280000400:6753) [ 8375.603929] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8377.424696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8387.062605] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 10:59:41 (1783954781) [ 8405.409323] Lustre: Failing over lustre-OST0000 [ 8405.985959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8405.987082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8406.015667] Lustre: Skipped 2 previous similar messages [ 8407.662709] Lustre: server umount lustre-OST0000 complete [ 8409.952723] LustreError: 150282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8409.980919] LustreError: 150282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8426.435972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8433.942817] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8445.752942] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8447.554261] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8459.313700] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 11:00:53 (1783954853) [ 8466.494546] Lustre: Failing over lustre-MDT0000 [ 8468.932188] Lustre: server umount lustre-MDT0000 complete [ 8476.923486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.848588] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6709 to 0x280000400:6785) [ 8477.850469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6886 to 0x240000400:6913) [ 8483.031348] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8493.570991] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 11:01:28 (1783954888) [ 8503.429620] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8506.581321] Lustre: Failing over lustre-OST0000 [ 8506.733578] Lustre: server umount lustre-OST0000 complete [ 8508.395206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8529.774743] LDISKFS-fs (dm-1): 3 truncates cleaned up [ 8529.776290] LDISKFS-fs (dm-1): recovery complete [ 8529.789147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8535.886533] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8547.972930] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8549.483968] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8559.440892] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 11:02:34 (1783954954) [ 8569.197585] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8573.149948] Lustre: Failing over lustre-OST0000 [ 8573.360470] Lustre: server umount lustre-OST0000 complete [ 8575.984984] LustreError: 42498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8576.011448] LustreError: 42498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 8596.598458] LDISKFS-fs (dm-1): 3 truncates cleaned up [ 8596.603241] LDISKFS-fs (dm-1): recovery complete [ 8596.613955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8598.842556] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.13@tcp inode [0x2000284a1:0x5:0x0] object 0x240000400:6915 extent [0-4194303]: client csum 53a08c53, server csum 4f679c0d [ 8604.944449] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8616.950640] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8618.856726] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8629.506052] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 11:03:43 (1783955023) [ 8638.536385] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8647.633147] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8654.392460] Lustre: Failing over lustre-MDT0000 [ 8654.860243] Lustre: server umount lustre-MDT0000 complete [ 8658.775209] LustreError: 5968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955055 with bad export cookie 10328443595428760311 [ 8669.154081] Lustre: Failing over lustre-OST0000 [ 8669.278815] Lustre: server umount lustre-OST0000 complete [ 8695.390359] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8695.403336] LDISKFS-fs (dm-0): recovery complete [ 8695.428708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8708.922650] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8719.318418] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6709 to 0x280000400:6817) [ 8732.161705] LDISKFS-fs (dm-1): 3 truncates cleaned up [ 8732.166252] LDISKFS-fs (dm-1): recovery complete [ 8732.182664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8739.671797] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8758.892123] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 11:05:53 (1783955153) [ 8771.932110] Lustre: Failing over lustre-OST0000 [ 8772.161484] Lustre: server umount lustre-OST0000 complete [ 8773.608880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8776.643397] Lustre: Failing over lustre-MDT0000 [ 8776.968406] Lustre: server umount lustre-MDT0000 complete [ 8794.915470] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783955175/real 1783955175] req@ffff95964402fb80 x1870603518894336/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783955191 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8794.946850] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 8794.964248] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.985374] Lustre: Skipped 10 previous similar messages [ 8795.105323] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8795.121624] LustreError: Skipped 5 previous similar messages [ 8797.142507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8804.327357] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f5600ba4430ad4f [ 8804.346897] Lustre: MGC192.168.202.113@tcp: Connection restored to 0@lo (at 0@lo) [ 8804.349657] Lustre: Skipped 13 previous similar messages [ 8804.881382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8804.891854] Lustre: Skipped 9 previous similar messages [ 8804.942694] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8804.964059] Lustre: Skipped 7 previous similar messages [ 8807.072274] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 8807.077546] Lustre: Skipped 7 previous similar messages [ 8807.133730] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6709 to 0x280000400:6849) [ 8810.817542] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8820.312451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8826.820503] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8829.586597] Lustre: lustre-OST0000: Denying connection for new client 6055c4ff-dc5c-4c95-904d-e60692e47661 (at 192.168.202.13@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 8829.624199] Lustre: Skipped 1 previous similar message [ 8850.284040] Lustre: lustre-OST0000: Denying connection for new client 6055c4ff-dc5c-4c95-904d-e60692e47661 (at 192.168.202.13@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:45 [ 8850.299747] Lustre: Skipped 3 previous similar messages [ 8886.109653] Lustre: lustre-OST0000: Denying connection for new client 6055c4ff-dc5c-4c95-904d-e60692e47661 (at 192.168.202.13@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:09 [ 8886.132112] Lustre: Skipped 6 previous similar messages [ 8895.500192] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 8895.505329] Lustre: 201707:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client c2ade49b-eb58-4a23-b26a-8c7edcaa73af@ [ 8895.520846] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 8895.567197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6964 to 0x240000400:6985) [ 8898.515492] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 58 sec [ 8909.573330] Lustre: DEBUG MARKER: free_before: 7646556 free_after: 7646556 [ 8917.866546] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 11:08:32 (1783955312) [ 8923.131878] Lustre: Failing over lustre-OST0001 [ 8923.282463] Lustre: server umount lustre-OST0001 complete [ 8927.093831] LustreError: 42508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8927.108996] LustreError: 42508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 8927.713697] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8944.416277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8946.063188] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8946.067811] Lustre: Skipped 8 previous similar messages [ 8951.779578] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8963.304769] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 11:09:18 (1783955358) [ 8968.572721] Lustre: Failing over lustre-OST0000 [ 8968.757212] Lustre: server umount lustre-OST0000 complete [ 8988.534141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8990.752697] LustreError: 204716:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 8990.757336] LustreError: 204716:0:(ldlm_lib.c:2904:target_recovery_thread()) Skipped 37 previous similar messages [ 8996.832148] Lustre: *** cfs_fail_loc=715, val=40*** [ 8998.399090] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9006.952389] Lustre: lustre-OST0000: Client 6055c4ff-dc5c-4c95-904d-e60692e47661 (at 192.168.202.13@tcp) reconnected, waiting for 2 clients in recovery for 1:24 [ 9013.217215] Lustre: *** cfs_fail_loc=715, val=40*** [ 9013.225796] Lustre: Skipped 1 previous similar message [ 9022.434584] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 2 clients in recovery for 1:09 [ 9022.457581] Lustre: Skipped 1 previous similar message [ 9028.580134] Lustre: *** cfs_fail_loc=715, val=40*** [ 9028.585304] Lustre: Skipped 1 previous similar message [ 9030.800348] LustreError: 204716:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9030.810444] LustreError: 204716:0:(ldlm_lib.c:2904:target_recovery_thread()) Skipped 77 previous similar messages [ 9039.167363] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9041.096620] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9051.771088] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 11:10:46 (1783955446) [ 9057.458627] Lustre: Failing over lustre-MDT0000 [ 9057.876387] Lustre: server umount lustre-MDT0000 complete [ 9078.028882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9084.781389] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (not set up) [ 9086.792717] LustreError: 206284:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 9091.129588] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9093.106602] Lustre: *** cfs_fail_loc=715, val=80*** [ 9093.113294] Lustre: Skipped 1 previous similar message [ 9103.195793] Lustre: lustre-MDT0000: Client 6055c4ff-dc5c-4c95-904d-e60692e47661 (at 192.168.202.13@tcp) reconnected, waiting for 1 clients in recovery for 0:53 [ 9103.212147] Lustre: Skipped 1 previous similar message [ 9109.472478] Lustre: *** cfs_fail_loc=715, val=80*** [ 9119.583276] Lustre: lustre-MDT0000: Client 6055c4ff-dc5c-4c95-904d-e60692e47661 (at 192.168.202.13@tcp) reconnected, waiting for 1 clients in recovery for 0:36 [ 9125.857709] Lustre: *** cfs_fail_loc=715, val=80*** [ 9135.966610] Lustre: lustre-MDT0000: Client 6055c4ff-dc5c-4c95-904d-e60692e47661 (at 192.168.202.13@tcp) reconnected, waiting for 1 clients in recovery for 0:20 [ 9142.240208] Lustre: *** cfs_fail_loc=715, val=80*** [ 9152.349717] Lustre: lustre-MDT0000: Client 6055c4ff-dc5c-4c95-904d-e60692e47661 (at 192.168.202.13@tcp) reconnected, waiting for 1 clients in recovery for 0:04 [ 9166.856157] LustreError: 206284:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9166.884515] Lustre: 206284:0:(ldlm_lib.c:2950:target_recovery_thread()) too long recovery - read logs [ 9166.908161] LustreError: dumping log to /tmp/lustre-log.1783955563.206284 [ 9167.243481] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6862 to 0x280000400:6881) [ 9167.245136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6999 to 0x240000400:7017) [ 9173.714039] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9174.935810] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9186.834359] Lustre: DEBUG MARKER: == replay-single test complete, duration 8849 sec ======== 11:13:01 (1783955581) [ 9188.422675] Lustre: DEBUG MARKER: === replay-single: start cleanup 11:13:03 (1783955583) === [ 9199.915448] Lustre: DEBUG MARKER: === replay-single: finish cleanup 11:13:14 (1783955594) === [ 9232.353864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9235.723764] Lustre: server umount lustre-MDT0000 complete [ 9245.925981] LustreError: 207230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955642 with bad export cookie 10328443595428772470 [ 9246.086355] Lustre: server umount lustre-OST0000 complete [ 9257.249733] Lustre: server umount lustre-OST0001 complete [ 9276.858474] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 9280.366493] Key type lgssc unregistered [ 9280.839484] LNet: 209132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9280.855424] LNetError: 209132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9280.882780] LNet: Removed LNI 192.168.202.113@tcp [ 9281.987191] Key type .llcrypt unregistered [ 9281.989476] Key type ._llcrypt unregistered