[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 590430482 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002371] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008017] pid_max: default: 32768 minimum: 301 [ 0.009105] LSM: Security Framework initializing [ 0.010067] Yama: becoming mindful. [ 0.011033] SELinux: Initializing. [ 0.012000] *** VALIDATE selinux *** [ 0.012000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.012000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.012000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.012000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.012000] *** VALIDATE tmpfs *** [ 0.012000] *** VALIDATE proc *** [ 0.012000] *** VALIDATE cgroup *** [ 0.012000] *** VALIDATE cgroup2 *** [ 0.012000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.012000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.012000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.012000] Spectre V2 : User space: Vulnerable [ 0.012000] Speculative Store Bypass: Vulnerable [ 0.012000] debug: unmapping init [mem 0xffffffff89a59000-0xffffffff89a60fff] [ 0.012000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.012665] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.013024] ... version: 2 [ 0.014016] ... bit width: 48 [ 0.015016] ... generic registers: 4 [ 0.016020] ... value mask: 0000ffffffffffff [ 0.017025] ... max period: 00007fffffffffff [ 0.018026] ... fixed-purpose events: 3 [ 0.019018] ... event mask: 000000070000000f [ 0.020347] rcu: Hierarchical SRCU implementation. [ 0.023584] smp: Bringing up secondary CPUs ... [ 0.024613] x86: Booting SMP configuration: [ 0.025043] .... node #0, CPUs: #1 #2 #3 [ 0.040017] smp: Brought up 1 node, 4 CPUs [ 0.042890] smpboot: Max logical packages: 1 [ 0.043018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.067015] node 0 deferred pages initialised in 21ms [ 0.072138] devtmpfs: initialized [ 0.073230] x86/mm: Memory block size: 128MB [ 0.077745] gcov: version magic: 0x41383552 [ 0.083644] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.092083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.099438] pinctrl core: initialized pinctrl subsystem [ 0.125197] [ 0.126007] ************************************************************* [ 0.129014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.133177] ** ** [ 0.136097] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.141021] ** ** [ 0.145015] ** This means that this kernel is built to expose internal ** [ 0.150013] ** IOMMU data structures, which may compromise security on ** [ 0.157025] ** your system. ** [ 0.163031] ** ** [ 0.170016] ** If you see this message and you are not debugging the ** [ 0.176014] ** kernel, report this immediately to your vendor! ** [ 0.183015] ** ** [ 0.189014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194017] ************************************************************* [ 0.206356] NET: Registered protocol family 16 [ 0.207404] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.208046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.210050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.215010] cpuidle: using governor menu [ 0.216659] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.219355] PCI: Using configuration type 1 for base access [ 0.222217] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.234455] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.236026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.241070] cryptd: max_cpu_qlen set to 1000 [ 0.245378] ACPI: Added _OSI(Module Device) [ 0.246020] ACPI: Added _OSI(Processor Device) [ 0.250010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.251017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.257553] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.279603] ACPI: Interpreter enabled [ 0.280074] ACPI: PM: (supports S0 S3 S4 S5) [ 0.281010] ACPI: Using IOAPIC for interrupt routing [ 0.282087] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.286453] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.296929] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.299044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.302014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.304061] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.308175] acpiphp: Slot [2] registered [ 0.309100] acpiphp: Slot [5] registered [ 0.311216] acpiphp: Slot [6] registered [ 0.312186] acpiphp: Slot [7] registered [ 0.315275] acpiphp: Slot [8] registered [ 0.317253] acpiphp: Slot [9] registered [ 0.320122] acpiphp: Slot [10] registered [ 0.323000] acpiphp: Slot [3] registered [ 0.324233] acpiphp: Slot [4] registered [ 0.326079] acpiphp: Slot [11] registered [ 0.327092] acpiphp: Slot [12] registered [ 0.329097] acpiphp: Slot [13] registered [ 0.330283] acpiphp: Slot [14] registered [ 0.332122] acpiphp: Slot [15] registered [ 0.335211] acpiphp: Slot [16] registered [ 0.336111] acpiphp: Slot [17] registered [ 0.338124] acpiphp: Slot [18] registered [ 0.339803] acpiphp: Slot [19] registered [ 0.342201] acpiphp: Slot [20] registered [ 0.343210] acpiphp: Slot [21] registered [ 0.346200] acpiphp: Slot [22] registered [ 0.347214] acpiphp: Slot [23] registered [ 0.349111] acpiphp: Slot [24] registered [ 0.350185] acpiphp: Slot [25] registered [ 0.352109] acpiphp: Slot [26] registered [ 0.354041] acpiphp: Slot [27] registered [ 0.355125] acpiphp: Slot [28] registered [ 0.356144] acpiphp: Slot [29] registered [ 0.358122] acpiphp: Slot [30] registered [ 0.360123] acpiphp: Slot [31] registered [ 0.361000] PCI host bridge to bus 0000:00 [ 0.362023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.364022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.367020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.369022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.371024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.375028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.378180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.383908] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.387926] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.403014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.408063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.409000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.411017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.415017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.418550] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.422499] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.427045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.430841] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.434759] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.448015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.454027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.460264] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.469019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.474020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.508016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.522381] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.531021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.539017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.572021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.576000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.600027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.617028] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.647030] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.666000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.679047] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.686030] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.711033] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.738349] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.795025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.809024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.839019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.874233] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.893170] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.904015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.920017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.928769] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.931357] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.933434] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.935387] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.937209] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.941088] iommu: Default domain type: Passthrough [ 0.943071] SCSI subsystem initialized [ 0.944214] ACPI: bus type USB registered [ 0.945237] usbcore: registered new interface driver usbfs [ 0.946097] usbcore: registered new interface driver hub [ 0.947208] usbcore: registered new device driver usb [ 0.948559] pps_core: LinuxPPS API ver. 1 registered [ 0.949022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.951081] PTP clock support registered [ 0.953154] EDAC MC: Ver: 3.0.0 [ 0.954213] PCI: Using ACPI for IRQ routing [ 0.956185] NetLabel: Initializing [ 0.956841] NetLabel: domain hash size = 128 [ 0.957005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.958053] NetLabel: unlabeled traffic allowed by default [ 0.959318] vgaarb: loaded [ 0.960329] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.961010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.965499] clocksource: Switched to clocksource kvm-clock [ 1.073424] VFS: Disk quotas dquot_6.6.0 [ 1.074582] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.076557] *** VALIDATE ramfs *** [ 1.078055] *** VALIDATE hugetlbfs *** [ 1.080230] pnp: PnP ACPI init [ 1.083329] pnp: PnP ACPI: found 6 devices [ 1.099805] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.101897] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.103088] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.104243] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.105719] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.107308] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.109202] NET: Registered protocol family 2 [ 1.110926] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.114221] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.118646] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.125303] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.129243] TCP: Hash tables configured (established 65536 bind 65536) [ 1.132157] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.135502] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.138873] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.141779] NET: Registered protocol family 1 [ 1.144410] RPC: Registered named UNIX socket transport module. [ 1.148051] RPC: Registered udp transport module. [ 1.149713] RPC: Registered tcp transport module. [ 1.151286] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.154169] NET: Registered protocol family 44 [ 1.155706] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.158045] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.160575] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.163245] PCI: CLS 0 bytes, default 64 [ 1.165654] Unpacking initramfs... [ 3.993453] debug: unmapping init [mem 0xffff9fc1fcc54000-0xffff9fc1fffbffff] [ 3.997383] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.999461] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.002510] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.555980] Initialise system trusted keyrings [ 4.557429] Key type blacklist registered [ 4.558874] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.567970] zbud: loaded [ 4.570976] *** VALIDATE nfs *** [ 4.572123] *** VALIDATE nfs4 *** [ 4.573480] pstore: using deflate compression [ 4.577558] Platform Keyring initialized [ 4.692630] NET: Registered protocol family 38 [ 4.694289] Key type asymmetric registered [ 4.695625] Asymmetric key parser 'x509' registered [ 4.697227] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.700442] io scheduler mq-deadline registered [ 4.701933] io scheduler kyber registered [ 4.703524] io scheduler bfq registered [ 4.705456] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.708705] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.711647] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.714993] ACPI: Power Button [PWRF] [ 4.722996] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.730404] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.750898] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.765896] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.800635] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.835237] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.869416] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.873641] Non-volatile memory driver v1.3 [ 4.874907] Linux agpgart interface v0.103 [ 4.905671] virtio_blk virtio1: [vda] 134592 512-byte logical blocks (68.9 MB/65.7 MiB) [ 4.908402] vda: detected capacity change from 0 to 68911104 [ 4.961935] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.971122] vdb: detected capacity change from 0 to 1073741824 [ 5.007789] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.010585] vdc: detected capacity change from 0 to 2621440000 [ 5.049984] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.052624] vdd: detected capacity change from 0 to 2621440000 [ 5.109805] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.112694] vde: detected capacity change from 0 to 4294967296 [ 5.139410] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.141992] vdf: detected capacity change from 0 to 4294967296 [ 5.159575] libphy: Fixed MDIO Bus: probed [ 5.169687] usbcore: registered new interface driver usbserial_generic [ 5.171641] usbserial: USB Serial support registered for generic [ 5.174443] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.179527] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.181237] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.183298] mousedev: PS/2 mouse device common for all mice [ 5.186378] rtc_cmos 00:05: RTC can wake from S4 [ 5.189426] rtc_cmos 00:05: registered as rtc0 [ 5.191159] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.194890] intel_pstate: CPU model not supported [ 5.195437] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.203578] hid: raw HID events driver (C) Jiri Kosina [ 5.206358] usbcore: registered new interface driver usbhid [ 5.208441] usbhid: USB HID core driver [ 5.209922] drop_monitor: Initializing network drop monitor service [ 5.213204] Initializing XFRM netlink socket [ 5.215201] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.218694] NET: Registered protocol family 10 [ 5.222810] Segment Routing with IPv6 [ 5.224262] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.224501] NET: Registered protocol family 17 [ 5.228495] mpls_gso: MPLS GSO support [ 5.236190] RAS: Correctable Errors collector initialized. [ 5.238225] AVX version of gcm_enc/dec engaged. [ 5.239548] AES CTR mode by8 optimization enabled [ 5.330712] sched_clock: Marking stable (5330598666, 0)->(7618310201, -2287711535) [ 5.334586] registered taskstats version 1 [ 5.336518] Loading compiled-in X.509 certificates [ 5.338531] zswap: loaded using pool lzo/zbud [ 5.366638] Key type big_key registered [ 5.380785] Key type encrypted registered [ 5.382584] ima: No TPM chip found, activating TPM-bypass! [ 5.384829] ima: Allocated hash algorithm: sha1 [ 5.386637] ima: No architecture policies found [ 5.388360] evm: Initialising EVM extended attributes: [ 5.389848] evm: security.selinux [ 5.390933] evm: security.ima [ 5.391985] evm: security.capability [ 5.393379] evm: HMAC attrs: 0x1 [ 5.395630] rtc_cmos 00:05: setting system clock to 2026-01-10 01:21:41 UTC (1768008101) [ 5.403598] debug: unmapping init [mem 0xffffffff8aa03000-0xffffffff8abfffff] [ 5.406731] debug: unmapping init [mem 0xffffffff89782000-0xffffffff89a58fff] [ 5.415169] Write protecting the kernel read-only data: 28672k [ 5.422304] debug: unmapping init [mem 0xffffffff87e03000-0xffffffff87ffffff] [ 5.427212] debug: unmapping init [mem 0xffffffff88714000-0xffffffff887fffff] [ 5.648438] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.667309] systemd[1]: Detected virtualization kvm. [ 5.670152] systemd[1]: Detected architecture x86-64. [ 5.674432] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.713950] systemd[1]: No hostname configured. [ 5.716853] systemd[1]: Set hostname to . [ 5.719913] random: systemd: uninitialized urandom read (16 bytes read) [ 5.726225] systemd[1]: Initializing machine ID from random generator. [ 5.933275] random: ln: uninitialized urandom read (6 bytes read) [ 6.311590] random: systemd: uninitialized urandom read (16 bytes read) [ 6.315691] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.338994] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 6.355927] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.741228] device-mapper: uevent: version 1.0.3 [ 8.752603] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.558895] virtio_net virtio0 ens2: renamed from eth0 [ 10.613829] random: fast init done [ 10.963968] scsi host0: ata_piix [ 11.163150] scsi host1: ata_piix [ 11.164657] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.181445] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.899643] random: crng init done [ 16.900897] random: 7 urandom warning(s) missed due to ratelimiting [ 18.828384] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.045437] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.536652] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.539737] SELinux: Disabled at runtime. [ 24.705169] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.732736] systemd[1]: Detected virtualization kvm. [ 24.739963] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.653247] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.665647] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.687294] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.699067] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.719412] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.751680] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.790759] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 27.731911] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 28.715457] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.289670] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.428689] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.088986] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.323952] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 36.259789] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 37.331684] NFS: Registering the id_resolver key type [ 37.332995] Key type id_resolver registered [ 37.339872] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg431-server login: [ 75.645937] libcfs: loading out-of-tree module taints kernel. [ 75.668605] Key type ._llcrypt registered [ 75.670551] Key type .llcrypt registered [ 75.725625] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 84.800895] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 85.500303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 85.505666] alg: No test for adler32 (adler32-zlib) [ 86.616302] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g8820be3 [ 87.071038] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 88.728092] Key type lgssc registered [ 89.480527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.677109] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 98.590428] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.369605] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.171370] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.242686] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 119.214434] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 125.865232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.897907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 125.908221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 127.035216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 127.056401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 127.102530] Lustre: lustre-MDT0000: new disk, initializing [ 127.173936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 127.192127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 129.121339] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 136.429194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.469921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 136.529206] Lustre: 6482:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 136.554579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 136.557403] Lustre: Skipped 1 previous similar message [ 136.601745] Lustre: lustre-MDT0001: new disk, initializing [ 136.644287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 136.666605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 136.670861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 138.863174] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 141.668082] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 146.220184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 146.250942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 146.366137] Lustre: lustre-OST0000: new disk, initializing [ 146.368192] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 146.400585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 149.341061] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 151.030199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 151.035040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 151.063071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 155.687398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 155.719546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 155.768684] Lustre: lustre-OST0001: new disk, initializing [ 155.771527] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 155.794648] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.753423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 157.756635] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 157.770897] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 158.165217] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.707891] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 169.778128] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 177.431238] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing check_logdir /tmp/testlogs/ [ 179.664265] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing yml_node [ 181.650111] Lustre: DEBUG MARKER: Client: 2.17.0.RC3 [ 182.841734] Lustre: DEBUG MARKER: MDS: 2.17.0.RC3 [ 183.929117] Lustre: DEBUG MARKER: OSS: 2.17.0.RC3 [ 184.596054] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Fri Jan 9 20:24:40 EST 2026 [ 194.678591] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 195.537726] Lustre: DEBUG MARKER: === replay-single: start setup 20:24:50 (1768008290) === [ 197.297904] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing check_config_client /mnt/lustre [ 208.676267] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 210.569924] Lustre: 13155:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 212.196209] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.964509] Lustre: DEBUG MARKER: === replay-single: finish setup 20:25:10 (1768008310) === [ 215.882380] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 20:25:11 (1768008311) [ 219.963636] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 220.841544] Lustre: Failing over lustre-MDT0000 [ 220.917208] LustreError: 14026:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 220.964661] Lustre: server umount lustre-MDT0000 complete [ 223.712825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 223.717397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 224.226063] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 224.247535] Lustre: Skipped 2 previous similar messages [ 228.973824] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 228.989347] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 234.084142] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 234.094294] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 238.037889] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 238.040321] LDISKFS-fs (dm-0): recovery complete [ 238.054454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.090232] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 238.197947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.211623] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 240.297176] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.685948] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 243.705522] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 245.763329] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 246.702385] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 252.299496] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 20:25:47 (1768008347) [ 253.446686] Lustre: Failing over lustre-OST0000 [ 253.474190] LustreError: 15453:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 253.483551] LustreError: 15453:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 253.513626] Lustre: server umount lustre-OST0000 complete [ 253.929517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 253.930763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.955544] LustreError: 8374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 253.963571] LustreError: 8374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 258.528531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 258.538581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 258.545574] LustreError: 8373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.556221] LustreError: 8373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 264.164192] LustreError: 8838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 264.173920] LustreError: 8838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 268.123711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 268.232073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 269.434306] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 269.654162] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 269.657174] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 269.664794] Lustre: Skipped 3 previous similar messages [ 271.325469] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.905939] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 276.821323] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 282.440769] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 20:26:17 (1768008377) [ 286.610679] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 287.854304] Lustre: Failing over lustre-MDT0000 [ 287.922405] LustreError: 17155:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 287.927048] LustreError: 17155:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 287.994916] Lustre: server umount lustre-MDT0000 complete [ 288.738757] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.739238] LustreError: 15174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 288.746178] Lustre: Skipped 2 previous similar messages [ 288.764370] LustreError: 15174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 303.968117] Lustre: 3627:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768008384/real 1768008384] req@ffff9fc257eec000 x1853890948626048/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1768008400 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 303.991152] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.077837] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 305.080344] LDISKFS-fs (dm-0): recovery complete [ 305.086634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 309.216499] LustreError: 7429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.224190] LustreError: 7429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 314.337493] LustreError: 3626:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc257f55180 x1853890948634496/t0(0) o250->MGC192.168.204.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 314.456732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 316.408802] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 317.617377] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 317.621668] Lustre: lustre-MDT0000: Denying connection for new client 0dbf1fd9-4f22-454e-959e-c2c180884ee2 (at 192.168.204.31@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 319.462900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 319.466440] Lustre: Skipped 1 previous similar message [ 322.660308] Lustre: lustre-MDT0000: Denying connection for new client 0dbf1fd9-4f22-454e-959e-c2c180884ee2 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 322.674446] Lustre: Skipped 1 previous similar message [ 327.779302] Lustre: lustre-MDT0000: Denying connection for new client 0dbf1fd9-4f22-454e-959e-c2c180884ee2 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 332.899774] Lustre: lustre-MDT0000: Denying connection for new client 0dbf1fd9-4f22-454e-959e-c2c180884ee2 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 338.018282] Lustre: lustre-MDT0000: Denying connection for new client 0dbf1fd9-4f22-454e-959e-c2c180884ee2 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 348.258858] Lustre: lustre-MDT0000: Denying connection for new client 0dbf1fd9-4f22-454e-959e-c2c180884ee2 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 348.269082] Lustre: Skipped 1 previous similar message [ 368.737848] Lustre: lustre-MDT0000: Denying connection for new client 0dbf1fd9-4f22-454e-959e-c2c180884ee2 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 368.746652] Lustre: Skipped 3 previous similar messages [ 387.500535] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 387.508185] Lustre: 17808:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f4de1a51-c47a-45c8-a806-5c0969371018@ [ 387.518987] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 387.529484] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 387.529579] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 387.535306] Lustre: Skipped 2 previous similar messages [ 387.550932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 387.551690] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 394.429149] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 20:28:09 (1768008489) [ 398.525863] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 399.769324] Lustre: Failing over lustre-MDT0000 [ 399.860728] LustreError: 18867:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 399.864367] LustreError: 18867:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 399.922654] Lustre: server umount lustre-MDT0000 complete [ 401.376899] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.377371] LustreError: 8399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 401.383914] Lustre: Skipped 4 previous similar messages [ 401.395830] LustreError: 8399:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 416.690939] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 416.692463] LDISKFS-fs (dm-0): recovery complete [ 416.696476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.759167] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 416.863044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.134661] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 420.376413] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 420.382161] Lustre: lustre-MDT0000: Denying connection for new client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 420.390822] Lustre: Skipped 3 previous similar messages [ 421.868504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 487.012321] Lustre: lustre-MDT0000: Denying connection for new client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:03 [ 487.024035] Lustre: Skipped 12 previous similar messages [ 490.504227] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 490.507318] Lustre: 19517:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0dbf1fd9-4f22-454e-959e-c2c180884ee2@ [ 490.514225] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 490.521824] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 490.524757] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 490.535066] Lustre: Skipped 2 previous similar messages [ 490.545627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 490.545652] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 496.937383] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 20:29:52 (1768008592) [ 500.725293] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 501.734209] Lustre: Failing over lustre-MDT0000 [ 501.808501] LustreError: 20574:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 501.810397] LustreError: 20574:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 501.841027] Lustre: server umount lustre-MDT0000 complete [ 502.384180] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.395709] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 503.776893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.780954] Lustre: Skipped 3 previous similar messages [ 517.340420] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 517.342148] LDISKFS-fs (dm-0): recovery complete [ 517.346434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.404828] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 517.502511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.729916] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 519.150170] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 522.726392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 522.741685] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 522.762039] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 522.762377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 524.290493] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 525.049815] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 529.360916] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 20:30:24 (1768008624) [ 532.700889] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 533.669842] Lustre: Failing over lustre-MDT0000 [ 533.764995] LustreError: 22394:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 533.769255] LustreError: 22394:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 533.801824] Lustre: server umount lustre-MDT0000 complete [ 538.081046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 538.081195] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.090696] Lustre: Skipped 3 previous similar messages [ 549.328191] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 549.329858] LDISKFS-fs (dm-0): recovery complete [ 549.333725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.390200] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 551.227391] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 553.575169] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 554.978524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 554.984974] Lustre: Skipped 3 previous similar messages [ 555.023243] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 555.051915] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 555.053254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 556.552234] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 557.186906] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 561.089310] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 20:30:56 (1768008656) [ 564.356625] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 565.227976] Lustre: Failing over lustre-MDT0000 [ 565.336170] LustreError: 24217:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 565.338641] LustreError: 24217:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 565.377370] Lustre: server umount lustre-MDT0000 complete [ 570.337376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 580.489905] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 580.492060] LDISKFS-fs (dm-0): recovery complete [ 580.497257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.556633] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 582.303699] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 584.290339] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 585.738619] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 585.761899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 585.761960] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:193) [ 587.185458] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 587.872757] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 591.856239] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 20:31:27 (1768008687) [ 595.039862] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 595.864850] Lustre: Failing over lustre-MDT0000 [ 595.936378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 595.940071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 595.942688] Lustre: Skipped 3 previous similar messages [ 596.041624] Lustre: server umount lustre-MDT0000 complete [ 611.143427] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 611.145668] LDISKFS-fs (dm-0): recovery complete [ 611.149526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.185019] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.317362] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 613.260371] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 616.475312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 616.475424] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 617.970028] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 618.657602] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 622.802112] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 20:31:58 (1768008718) [ 625.852458] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 626.666078] Lustre: Failing over lustre-MDT0000 [ 626.779203] LustreError: 27861:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 626.781690] LustreError: 27861:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 626.802509] Lustre: server umount lustre-MDT0000 complete [ 631.777376] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.777868] LustreError: 24333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.778210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 631.783161] Lustre: Skipped 11 previous similar messages [ 631.789542] LustreError: 24333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 58 previous similar messages [ 641.826232] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 641.829085] LDISKFS-fs (dm-0): recovery complete [ 641.834718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.992552] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 643.632037] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 645.730152] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 645.740818] Lustre: Skipped 1 previous similar message [ 647.139362] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 647.142319] Lustre: Skipped 11 previous similar messages [ 647.166472] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 647.171356] Lustre: Skipped 1 previous similar message [ 647.187953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 647.188112] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 648.700713] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 649.385266] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 653.333821] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 20:32:28 (1768008748) [ 653.744355] Lustre: *** cfs_fail_loc=13b, val=315*** [ 653.746091] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 653.747821] LustreError: 6493:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc25cb21180 x1853890939181952/t38654705666(0) o35->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:345/0 lens 392/456 e 0 to 0 dl 1768008760 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 657.761982] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 658.482335] Lustre: Failing over lustre-MDT0000 [ 658.661232] Lustre: server umount lustre-MDT0000 complete [ 673.675660] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 673.677284] LDISKFS-fs (dm-0): recovery complete [ 673.683712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.750648] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.755840] LustreError: Skipped 1 previous similar message [ 673.840749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 673.843470] Lustre: Skipped 4 previous similar messages [ 673.863959] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 675.370311] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 678.904713] Lustre: 6493:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc2790a8380 x1853890939181952/t38654705666(0) o35->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:371/0 lens 392/456 e 0 to 0 dl 1768008786 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 678.920266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 678.920532] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 680.472541] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 681.145289] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 685.165892] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 20:33:00 (1768008780) [ 688.775573] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 689.652821] Lustre: Failing over lustre-MDT0000 [ 689.815535] Lustre: server umount lustre-MDT0000 complete [ 694.241995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 694.245183] LustreError: Skipped 1 previous similar message [ 705.024975] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 705.026787] LDISKFS-fs (dm-0): recovery complete [ 705.033238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 705.183805] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 706.734763] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 710.668374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 710.668599] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 712.149180] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 712.845414] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 716.756710] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 20:33:32 (1768008812) [ 720.127532] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 720.523252] Lustre: *** cfs_fail_loc=114, val=0*** [ 721.816512] Lustre: Failing over lustre-MDT0000 [ 721.911642] LustreError: 33474:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 721.914853] LustreError: 33474:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 721.981407] Lustre: server umount lustre-MDT0000 complete [ 737.363323] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 737.365156] LDISKFS-fs (dm-0): recovery complete [ 737.369240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.529826] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 737.890053] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 737.893487] Lustre: Skipped 2 previous similar messages [ 739.078064] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 742.919163] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 742.929949] Lustre: Skipped 2 previous similar messages [ 742.960613] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 742.962163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 744.573441] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 745.287943] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 749.636945] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 20:34:04 (1768008844) [ 753.408279] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 753.818504] Lustre: *** cfs_fail_loc=128, val=0*** [ 755.069139] Lustre: Failing over lustre-MDT0000 [ 755.269484] Lustre: server umount lustre-MDT0000 complete [ 758.241976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 758.246654] LustreError: Skipped 1 previous similar message [ 770.334880] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 770.336823] LDISKFS-fs (dm-0): recovery complete [ 770.341159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.390475] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 770.394460] LustreError: Skipped 2 previous similar messages [ 770.498045] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 772.076476] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 775.652123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 775.655361] Lustre: Skipped 15 previous similar messages [ 775.691251] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 775.691367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 777.342139] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 778.015907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 782.243695] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 20:34:37 (1768008877) [ 785.402598] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 786.331069] Lustre: Failing over lustre-MDT0000 [ 786.527450] Lustre: server umount lustre-MDT0000 complete [ 791.008507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.013190] Lustre: Skipped 17 previous similar messages [ 801.929813] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 801.931255] LDISKFS-fs (dm-0): recovery complete [ 801.936657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.081933] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 803.761498] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 807.519185] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 807.519370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 809.063656] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 809.861245] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 814.198296] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 20:35:09 (1768008909) [ 817.881446] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 818.970992] Lustre: Failing over lustre-MDT0000 [ 819.101449] Lustre: server umount lustre-MDT0000 complete [ 834.291989] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 834.293786] LDISKFS-fs (dm-0): recovery complete [ 834.298473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.442948] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 835.966645] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 839.748605] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 839.748719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 841.289305] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 842.129920] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 846.651075] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 20:35:41 (1768008941) [ 849.724316] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 853.608956] Lustre: Failing over lustre-MDT0000 [ 853.772597] LustreError: 40855:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 853.776587] LustreError: 40855:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 853.846758] Lustre: server umount lustre-MDT0000 complete [ 869.264291] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 869.266051] LDISKFS-fs (dm-0): recovery complete [ 869.271179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.010134] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 871.014640] Lustre: Skipped 3 previous similar messages [ 871.065026] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 875.514207] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 875.517797] Lustre: Skipped 3 previous similar messages [ 875.536298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 875.537239] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 876.891773] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 877.516224] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 889.767264] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 20:36:25 (1768008985) [ 892.820521] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 893.601669] Lustre: Failing over lustre-MDT0000 [ 893.797434] Lustre: server umount lustre-MDT0000 complete [ 894.945405] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 894.953908] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 114 previous similar messages [ 908.739551] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 908.741183] LDISKFS-fs (dm-0): recovery complete [ 908.745444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.788500] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 908.791759] LustreError: Skipped 3 previous similar messages [ 908.889053] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 908.891752] Lustre: Skipped 1 previous similar message [ 910.321182] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 913.921126] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 913.921132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 915.413385] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 916.095878] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 922.169693] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 20:36:57 (1768009017) [ 925.383277] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 926.168973] Lustre: Failing over lustre-MDT0000 [ 926.349252] Lustre: server umount lustre-MDT0000 complete [ 929.249429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 929.252635] LustreError: Skipped 3 previous similar messages [ 941.287642] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 941.293041] LDISKFS-fs (dm-0): recovery complete [ 941.299563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 941.460345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 941.463381] Lustre: Skipped 7 previous similar messages [ 943.338253] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 946.694378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 946.694468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 948.034301] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 948.633714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 952.453026] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 20:37:27 (1768009047) [ 955.308881] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 956.113243] Lustre: Failing over lustre-MDT0000 [ 956.289020] Lustre: server umount lustre-MDT0000 complete [ 971.090291] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 971.092266] LDISKFS-fs (dm-0): recovery complete [ 971.098517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 972.801289] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 976.397456] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 976.397846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 977.718235] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 978.336915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 982.154394] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 20:37:57 (1768009077) [ 985.106772] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 985.884252] Lustre: Failing over lustre-MDT0000 [ 986.050875] Lustre: server umount lustre-MDT0000 complete [ 1000.614334] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1000.616367] LDISKFS-fs (dm-0): recovery complete [ 1000.620029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.190400] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1006.102398] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1006.102398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1007.407247] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1008.057462] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1011.984300] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 20:38:27 (1768009107) [ 1014.943980] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1015.746754] Lustre: Failing over lustre-MDT0000 [ 1015.897047] Lustre: server umount lustre-MDT0000 complete [ 1030.606193] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1030.607891] LDISKFS-fs (dm-0): recovery complete [ 1030.612986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.339234] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1035.747534] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1035.749424] Lustre: Skipped 31 previous similar messages [ 1035.794728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1035.794948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1037.377180] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1038.086592] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1042.105991] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 20:38:57 (1768009137) [ 1044.956657] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1045.686131] Lustre: Failing over lustre-MDT0000 [ 1045.880717] Lustre: server umount lustre-MDT0000 complete [ 1060.488303] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1060.490130] LDISKFS-fs (dm-0): recovery complete [ 1060.494438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.639460] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1060.641908] Lustre: Skipped 4 previous similar messages [ 1062.141922] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1065.990564] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1065.990575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1067.330361] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1068.037940] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1071.839878] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 20:39:27 (1768009167) [ 1074.778792] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1075.640165] Lustre: Failing over lustre-MDT0000 [ 1075.781943] Lustre: server umount lustre-MDT0000 complete [ 1076.192756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.199171] Lustre: Skipped 36 previous similar messages [ 1090.716871] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1090.718607] LDISKFS-fs (dm-0): recovery complete [ 1090.729292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.387299] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1096.209804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1096.210799] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1097.553121] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1098.264598] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1102.277074] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 20:39:57 (1768009197) [ 1105.349506] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1106.138352] Lustre: Failing over lustre-MDT0000 [ 1106.339050] Lustre: server umount lustre-MDT0000 complete [ 1121.332605] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1121.334224] LDISKFS-fs (dm-0): recovery complete [ 1121.339147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1122.995101] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1126.912905] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1126.913236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1128.339264] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1128.990790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1132.762827] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 20:40:28 (1768009228) [ 1135.522526] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1136.227903] Lustre: Failing over lustre-MDT0000 [ 1136.361272] LustreError: 57271:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1136.365381] LustreError: 57271:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages [ 1136.390158] Lustre: server umount lustre-MDT0000 complete [ 1150.975431] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1150.977266] LDISKFS-fs (dm-0): recovery complete [ 1150.981836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.571237] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1152.610061] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1152.615843] Lustre: Skipped 8 previous similar messages [ 1156.587299] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1156.590785] Lustre: Skipped 8 previous similar messages [ 1156.606247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 1156.606307] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 1157.928083] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1158.551874] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1162.462432] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 20:40:57 (1768009257) [ 1165.603829] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1166.291104] Lustre: Failing over lustre-MDT0000 [ 1166.480708] Lustre: server umount lustre-MDT0000 complete [ 1181.110621] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1181.112608] LDISKFS-fs (dm-0): recovery complete [ 1181.117851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.171967] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1181.175583] LustreError: Skipped 8 previous similar messages [ 1182.873883] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1186.301636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 1186.302044] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:897) [ 1187.623475] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1188.301740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1192.083283] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 20:41:27 (1768009287) [ 1195.028423] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1195.792176] Lustre: Failing over lustre-MDT0000 [ 1195.999590] Lustre: server umount lustre-MDT0000 complete [ 1196.513095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1196.515736] LustreError: Skipped 8 previous similar messages [ 1210.533233] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1210.535020] LDISKFS-fs (dm-0): recovery complete [ 1210.538926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1212.079360] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1216.005552] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1216.005552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1217.273299] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1217.913151] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1221.718562] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 20:41:57 (1768009317) [ 1224.510626] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1225.210377] Lustre: Failing over lustre-MDT0000 [ 1225.335184] Lustre: server umount lustre-MDT0000 complete [ 1239.775251] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1239.777036] LDISKFS-fs (dm-0): recovery complete [ 1239.785224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.315070] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1245.186410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 1245.187835] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 1246.463784] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1247.060734] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1250.614037] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 20:42:26 (1768009346) [ 1253.270234] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1253.988042] Lustre: Failing over lustre-MDT0000 [ 1254.145356] Lustre: server umount lustre-MDT0000 complete [ 1268.525815] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1268.527504] LDISKFS-fs (dm-0): recovery complete [ 1268.531119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1269.994595] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1273.850664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:993) [ 1273.851504] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 1275.122696] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1275.708876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1279.371224] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 20:42:54 (1768009374) [ 1282.119502] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1282.849935] Lustre: Failing over lustre-MDT0000 [ 1282.998058] Lustre: server umount lustre-MDT0000 complete [ 1297.269396] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1297.270781] LDISKFS-fs (dm-0): recovery complete [ 1297.275142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.771837] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1302.535433] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1302.535468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1303.780831] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1304.377301] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1307.978081] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 20:43:23 (1768009403) [ 1310.698462] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1311.424670] Lustre: Failing over lustre-MDT0000 [ 1311.642667] Lustre: server umount lustre-MDT0000 complete [ 1325.858745] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1325.860240] LDISKFS-fs (dm-0): recovery complete [ 1325.863379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.978896] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1325.982098] Lustre: Skipped 8 previous similar messages [ 1327.246156] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1331.206663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1331.206736] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1332.415231] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1333.018637] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1336.481295] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:43:52 (1768009432) [ 1339.034710] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1339.697633] Lustre: Failing over lustre-MDT0000 [ 1339.831642] Lustre: server umount lustre-MDT0000 complete [ 1353.869935] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1353.871599] LDISKFS-fs (dm-0): recovery complete [ 1353.874949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.296918] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1359.355072] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 1359.355083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 1360.476443] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1360.991328] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1364.359609] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 20:44:19 (1768009459) [ 1365.663377] Lustre: 71494:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a9792733-3f58-4d18-919d-c5231f8fa15b at adminstrative request [ 1369.146878] Lustre: Failing over lustre-MDT0000 [ 1369.330786] Lustre: server umount lustre-MDT0000 complete [ 1382.080812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.499582] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1387.507054] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 1387.507325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1388.700029] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1389.271320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1391.614875] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1394.321262] Lustre: DEBUG MARKER: before 3156, after 3156 [ 1396.581978] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 20:44:52 (1768009492) [ 1396.934481] Lustre: 73338:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a9792733-3f58-4d18-919d-c5231f8fa15b at adminstrative request [ 1401.706866] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 20:44:57 (1768009497) [ 1404.355302] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1405.054388] Lustre: Failing over lustre-MDT0000 [ 1405.298569] Lustre: server umount lustre-MDT0000 complete [ 1407.970877] LustreError: 6495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.978952] LustreError: 6495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 252 previous similar messages [ 1419.585650] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1419.587525] LDISKFS-fs (dm-0): recovery complete [ 1419.591038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.237081] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1424.920123] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 1424.920139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 1426.134509] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1426.690158] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1430.198276] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:45:25 (1768009525) [ 1432.854045] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1433.608763] Lustre: Failing over lustre-MDT0000 [ 1433.812879] Lustre: server umount lustre-MDT0000 complete [ 1448.472514] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1448.474143] LDISKFS-fs (dm-0): recovery complete [ 1448.477222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.999525] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1454.094573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1454.094574] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1455.397321] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1456.063469] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1460.072631] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 20:45:55 (1768009555) [ 1462.968356] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1463.678142] Lustre: Failing over lustre-MDT0000 [ 1463.829068] Lustre: server umount lustre-MDT0000 complete [ 1478.518153] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1478.519991] LDISKFS-fs (dm-0): recovery complete [ 1478.524843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.634859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1478.636935] Lustre: Skipped 17 previous similar messages [ 1480.000217] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1483.772046] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1483.772110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1485.038953] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1485.662250] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1489.348451] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 20:46:24 (1768009584) [ 1492.114294] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1492.815967] Lustre: Failing over lustre-MDT0000 [ 1493.047851] Lustre: server umount lustre-MDT0000 complete [ 1507.401290] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1507.402827] LDISKFS-fs (dm-0): recovery complete [ 1507.405967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.877355] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1512.950913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1512.950922] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1514.066253] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1514.566966] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1517.861426] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:46:53 (1768009613) [ 1520.307639] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1521.016369] Lustre: Failing over lustre-MDT0000 [ 1521.183414] Lustre: server umount lustre-MDT0000 complete [ 1535.527479] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1535.529011] LDISKFS-fs (dm-0): recovery complete [ 1535.533482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1537.035732] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1541.112456] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 1541.112457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 1542.372080] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1542.998783] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1546.681099] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 20:47:22 (1768009642) [ 1549.479234] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1550.297402] Lustre: Failing over lustre-MDT0000 [ 1550.445541] Lustre: server umount lustre-MDT0000 complete [ 1564.649647] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1564.651308] LDISKFS-fs (dm-0): recovery complete [ 1564.656619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.058737] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1569.765555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1569.768122] Lustre: Skipped 71 previous similar messages [ 1569.805644] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 1569.805663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 1570.963980] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1571.491833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1574.938130] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:47:50 (1768009670) [ 1577.396747] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1578.066102] Lustre: Failing over lustre-MDT0000 [ 1578.204087] Lustre: server umount lustre-MDT0000 complete [ 1592.173157] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1592.174515] LDISKFS-fs (dm-0): recovery complete [ 1592.177303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1593.521026] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1597.442135] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 1597.442170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 1598.630641] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1599.185260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1602.553955] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 20:48:18 (1768009698) [ 1605.066612] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1605.787665] Lustre: Failing over lustre-MDT0000 [ 1605.997339] Lustre: server umount lustre-MDT0000 complete [ 1607.648645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1607.653502] Lustre: Skipped 72 previous similar messages [ 1619.960922] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1619.962713] LDISKFS-fs (dm-0): recovery complete [ 1619.966451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.307190] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1625.593275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 1625.593276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 1626.666833] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1627.239114] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1630.676876] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:48:46 (1768009726) [ 1633.269037] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1633.927123] Lustre: Failing over lustre-MDT0000 [ 1634.064472] Lustre: server umount lustre-MDT0000 complete [ 1648.387308] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1648.389109] LDISKFS-fs (dm-0): recovery complete [ 1648.393856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1649.904181] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1653.761124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 1653.761139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 1654.991252] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1655.660478] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1659.446787] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:49:14 (1768009754) [ 1662.631590] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1663.432936] Lustre: Failing over lustre-MDT0000 [ 1663.543943] LustreError: 90419:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1663.545953] LustreError: 90419:0:(obd_class.h:479:obd_check_dev()) Skipped 143 previous similar messages [ 1663.570751] Lustre: server umount lustre-MDT0000 complete [ 1677.969443] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1677.971660] LDISKFS-fs (dm-0): recovery complete [ 1677.976876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.545373] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1679.971973] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1679.974608] Lustre: Skipped 17 previous similar messages [ 1683.433726] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1683.438365] Lustre: Skipped 17 previous similar messages [ 1683.455832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 1683.455843] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 1684.644717] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1685.194541] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1688.745676] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 20:49:44 (1768009784) [ 1691.518967] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1692.235290] Lustre: Failing over lustre-MDT0000 [ 1692.423192] Lustre: server umount lustre-MDT0000 complete [ 1706.794130] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1706.796295] LDISKFS-fs (dm-0): recovery complete [ 1706.799602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.835368] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1706.838055] LustreError: Skipped 17 previous similar messages [ 1708.264320] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1712.120331] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 1712.120335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 1713.273782] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1713.890426] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1717.516249] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 20:50:13 (1768009813) [ 1717.868129] Lustre: 93618:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a9792733-3f58-4d18-919d-c5231f8fa15b at adminstrative request [ 1722.770218] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 20:50:18 (1768009818) [ 1725.067769] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1725.727119] Lustre: Failing over lustre-MDT0000 [ 1725.920924] Lustre: server umount lustre-MDT0000 complete [ 1727.457405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1727.460865] LustreError: Skipped 17 previous similar messages [ 1730.448472] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1730.450983] LDISKFS-fs (dm-0): recovery complete [ 1730.455633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.573532] Lustre: lustre-MDT0000: Aborting client recovery [ 1730.575863] LustreError: 94897:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1730.580088] Lustre: 94930:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1730.583512] Lustre: 94930:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a9792733-3f58-4d18-919d-c5231f8fa15b@ [ 1730.588831] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1730.592287] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1730.596836] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1730.617985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1505) [ 1730.617990] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 1732.044217] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1735.649621] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1744.135236] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 20:50:39 (1768009839) [ 1746.991714] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1747.656064] Lustre: Failing over lustre-MDT0000 [ 1747.840973] Lustre: server umount lustre-MDT0000 complete [ 1752.659062] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1752.660558] LDISKFS-fs (dm-0): recovery complete [ 1752.663921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1752.794024] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1752.826496] Lustre: lustre-MDT0000: Aborting client recovery [ 1752.828738] LustreError: 96742:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1752.832080] Lustre: 96775:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1752.836694] Lustre: 96775:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 1752.840762] Lustre: 96775:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1752.845343] Lustre: 96775:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1752.849311] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1752.855371] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 1752.862661] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 1752.890342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1537) [ 1752.890347] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1537) [ 1754.411894] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1758.177574] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1764.294390] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1764.296229] Lustre: Skipped 1 previous similar message [ 1766.728878] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 20:51:02 (1768009862) [ 1769.667706] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1770.415183] Lustre: Failing over lustre-MDT0000 [ 1770.505351] Lustre: server umount lustre-MDT0000 complete [ 1774.993540] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1774.995726] LDISKFS-fs (dm-0): recovery complete [ 1775.000279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1775.132654] Lustre: lustre-MDT0000: Aborting client recovery [ 1775.134488] LustreError: 98590:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1775.137312] Lustre: 98624:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1775.141496] Lustre: 98624:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 1775.143746] Lustre: 98624:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1775.147018] Lustre: 98624:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1775.149264] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1775.152466] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 1775.158765] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 1775.181127] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 1775.185809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 1776.376223] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1780.195832] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1787.604412] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 20:51:23 (1768009883) [ 1787.924127] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1787.925620] LustreError: 6491:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc25cb20000 x1853890940100352/t201863462916(0) o36->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:725/0 lens 512/456 e 0 to 0 dl 1768009895 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 1790.531731] Lustre: Failing over lustre-MDT0000 [ 1790.621516] Lustre: server umount lustre-MDT0000 complete [ 1793.471280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.623635] Lustre: lustre-MDT0000: Aborting client recovery [ 1793.625588] LustreError: 99897:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1793.628331] Lustre: 99930:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1793.631524] Lustre: 99930:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 1793.634492] Lustre: 99930:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1793.639124] Lustre: 99930:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1793.642313] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1793.645724] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 1793.652398] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 1793.672118] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1601) [ 1793.672128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 1794.911655] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1798.626670] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1807.116067] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 1807.651815] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 20:51:43 (1768009903) [ 1810.297209] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1811.193240] Lustre: Failing over lustre-MDT0000 [ 1811.342144] Lustre: server umount lustre-MDT0000 complete [ 1815.863244] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1815.865766] LDISKFS-fs (dm-0): recovery complete [ 1815.869521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.032876] Lustre: lustre-MDT0000: Aborting client recovery [ 1816.035102] LustreError: 101840:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1816.038977] Lustre: 101873:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1816.042523] Lustre: 101873:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 1816.045164] Lustre: 101873:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1816.049707] Lustre: 101873:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1816.053051] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1816.056857] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 1816.063774] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 1816.088664] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1633) [ 1816.088671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 1817.450435] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1821.153763] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1829.045875] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:52:04 (1768009924) [ 1837.659626] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1838.346976] Lustre: Failing over lustre-MDT0000 [ 1838.543033] Lustre: server umount lustre-MDT0000 complete [ 1852.530011] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1852.531654] LDISKFS-fs (dm-0): recovery complete [ 1852.535789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.715358] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1852.717481] Lustre: Skipped 28 previous similar messages [ 1853.883411] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1858.051824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 1858.051837] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 1859.164627] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1859.670564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1867.418656] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:52:43 (1768009963) [ 1872.932739] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1875.446025] Lustre: Failing over lustre-MDT0000 [ 1875.644965] Lustre: server umount lustre-MDT0000 complete [ 1889.603778] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1889.605872] LDISKFS-fs (dm-0): recovery complete [ 1889.609316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.985221] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1895.465821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 1895.465861] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 1896.593026] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1897.113589] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1904.636917] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 20:53:20 (1768010000) [ 1905.284299] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1905.572302] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1905.574120] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1907.444582] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 20:53:23 (1768010003) [ 1913.074607] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1916.104159] Lustre: Failing over lustre-OST0000 [ 1916.157452] Lustre: server umount lustre-OST0000 complete [ 1930.119170] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 1930.120351] LDISKFS-fs (dm-2): recovery complete [ 1930.122731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1931.948380] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1977.208961] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 20:54:32 (1768010072) [ 1979.825481] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1980.732282] Lustre: Failing over lustre-MDT0000 [ 1980.927945] Lustre: server umount lustre-MDT0000 complete [ 1995.007606] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1995.008793] LDISKFS-fs (dm-0): recovery complete [ 1995.010963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.339645] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2000.378228] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 2000.378256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 2001.415147] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2001.900625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2015.371052] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 20:55:10 (1768010110) [ 2016.736124] Lustre: 110927:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768010096/real 1768010096] req@ffff9fc146ae1c00 x1853890950360960/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1768010112 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-1-0.0' uid:0 gid:0 projid:4294967295 [ 2016.742957] LustreError: 110927:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2016.744028] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2016.902957] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2017.760429] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 2022.368100] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2022.371132] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnecting [ 2022.922310] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2028.000082] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2028.003112] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnecting [ 2028.563793] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2033.632127] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2033.635194] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnecting [ 2034.228464] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2039.264101] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2039.836235] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2044.896066] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2044.898157] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnecting [ 2044.900267] Lustre: Skipped 1 previous similar message [ 2051.095706] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2051.097542] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 2056.160082] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2056.163538] LustreError: 24333:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 2061.792165] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnecting [ 2061.795942] Lustre: Skipped 2 previous similar messages [ 2068.011084] LustreError: 6490:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2068.013973] LustreError: 6490:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 2073.056113] LustreError: 6490:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2073.059869] LustreError: 6490:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 2075.666156] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 20:56:11 (1768010171) [ 2076.260513] LustreError: 6490:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2086.497933] Lustre: lustre-MDT0000: Export ffff9fc1446d0800 already connecting from 192.168.204.31@tcp [ 2091.618178] Lustre: lustre-MDT0000: Export ffff9fc1446d0800 already connecting from 192.168.204.31@tcp [ 2096.737778] Lustre: lustre-MDT0000: Export ffff9fc1446d0800 already connecting from 192.168.204.31@tcp [ 2101.857785] Lustre: lustre-MDT0000: Export ffff9fc1446d0800 already connecting from 192.168.204.31@tcp [ 2101.861352] Lustre: Skipped 1 previous similar message [ 2106.977703] Lustre: lustre-MDT0000: Export ffff9fc1446d0800 already connecting from 192.168.204.31@tcp [ 2116.304107] LustreError: 6490:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2116.307275] Lustre: 6490:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9fc281b20000 x1853890942726912/t0(0) o38->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1768010192 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2117.218099] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnecting [ 2117.221240] Lustre: Skipped 3 previous similar messages [ 2117.222525] LustreError: 15174:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2142.817745] Lustre: lustre-MDT0000: Export ffff9fc1446d0800 already connecting from 192.168.204.31@tcp [ 2142.819928] Lustre: Skipped 1 previous similar message [ 2157.264142] LustreError: 15174:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2157.267383] Lustre: 15174:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9fc281b3dc00 x1853890942732160/t0(0) o38->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1768010233 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2158.177619] LustreError: 15174:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2183.777658] Lustre: lustre-MDT0000: Export ffff9fc1446d0800 already connecting from 192.168.204.31@tcp [ 2183.780761] Lustre: Skipped 2 previous similar messages [ 2198.224112] LustreError: 15174:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2198.226829] Lustre: 15174:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9fc25c421500 x1853890942735232/t0(0) o38->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1768010274 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2199.137520] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnecting [ 2199.139940] Lustre: Skipped 1 previous similar message [ 2199.141442] LustreError: 15174:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2224.737658] Lustre: lustre-MDT0000: Export ffff9fc1446d0800 already connecting from 192.168.204.31@tcp [ 2224.739837] Lustre: Skipped 2 previous similar messages [ 2239.184109] LustreError: 15174:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2239.186153] Lustre: 15174:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9fc2819d3b80 x1853890942738304/t0(0) o38->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1768010315 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2240.097866] LustreError: 6489:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2280.144099] LustreError: 6489:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2280.146183] Lustre: 6489:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9fc2819d1f80 x1853890942741376/t0(0) o38->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1768010356 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2281.057726] LustreError: 24333:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2286.368080] LustreError: 24333:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout interrupted [ 2288.041402] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 20:59:43 (1768010383) [ 2290.373018] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2291.433637] Lustre: Failing over lustre-MDT0000 [ 2291.490501] LustreError: 115009:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2291.492854] LustreError: 115009:0:(obd_class.h:479:obd_check_dev()) Skipped 83 previous similar messages [ 2291.522243] Lustre: server umount lustre-MDT0000 complete [ 2292.192568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.196511] Lustre: Skipped 48 previous similar messages [ 2292.198274] LustreError: 7429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.204273] LustreError: 7429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 232 previous similar messages [ 2295.347603] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2295.348878] LDISKFS-fs (dm-0): recovery complete [ 2295.351717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2295.446684] Lustre: *** cfs_fail_loc=712, val=0*** [ 2295.448942] LustreError: 40277:0:(service.c:1389:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9fc25c426a00 x1853890950501248/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2295.462131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.464728] Lustre: Skipped 17 previous similar messages [ 2295.498215] Lustre: lustre-MDT0000: Aborting client recovery [ 2295.500451] LustreError: 115652:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2295.504710] Lustre: 115685:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2295.508733] Lustre: 115685:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 2295.511329] Lustre: 115685:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2295.514053] Lustre: 115685:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2295.517284] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2295.519991] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2295.525284] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 2295.547448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 2295.547448] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 2296.724629] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2300.897181] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2300.905814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2300.908116] Lustre: Skipped 60 previous similar messages [ 2306.334800] Lustre: Failing over lustre-MDT0000 [ 2306.464396] Lustre: server umount lustre-MDT0000 complete [ 2319.029444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.071280] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.074868] LustreError: Skipped 9 previous similar messages [ 2320.328435] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2322.017709] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2322.020504] Lustre: Skipped 5 previous similar messages [ 2324.457827] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2324.460266] Lustre: Skipped 5 previous similar messages [ 2324.474362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 2324.474366] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 2325.542317] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2326.027387] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2329.511497] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 21:00:25 (1768010425) [ 2329.549470] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnecting [ 2329.553098] Lustre: Skipped 2 previous similar messages [ 2332.487267] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 21:00:28 (1768010428) [ 2332.791826] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2332.793983] LustreError: 6499:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc147401880 x1853890942797696/t0(0) o700->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:514/0 lens 264/248 e 0 to 0 dl 1768010439 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 2349.128640] Lustre: Failing over lustre-MDT0000 [ 2349.282172] Lustre: server umount lustre-MDT0000 complete [ 2350.048823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2350.051274] LustreError: Skipped 15 previous similar messages [ 2361.861807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.144540] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2367.478331] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 2367.478352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 2368.564297] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2369.098907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2372.988514] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 21:01:08 (1768010468) [ 2373.764470] Lustre: Failing over lustre-OST0000 [ 2375.879324] Lustre: server umount lustre-OST0000 complete [ 2388.490803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2390.215099] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2393.092281] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2393.571086] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2458.717556] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 21:02:34 (1768010554) [ 2461.200413] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2461.886237] Lustre: Failing over lustre-MDT0000 [ 2462.108335] Lustre: server umount lustre-MDT0000 complete [ 2476.111699] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2476.113052] LDISKFS-fs (dm-0): recovery complete [ 2476.115644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.247068] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2476.250384] Lustre: Skipped 9 previous similar messages [ 2477.600909] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2481.722678] Lustre: *** cfs_fail_loc=216, val=0*** [ 2481.722732] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 2481.724503] LustreError: 121462:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2482.784461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 2544.273353] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 21:03:59 (1768010639) [ 2552.302044] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 21:04:07 (1768010647) [ 2553.377100] Lustre: Failing over lustre-MDT0000 [ 2553.592271] Lustre: server umount lustre-MDT0000 complete [ 2566.617711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.132788] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2571.750408] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2571.751833] LustreError: 123025:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc278fb1880 x1853890942914304/t0(0) o101->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:38/0 lens 328/344 e 0 to 0 dl 1768010718 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2627.681753] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 1:06 [ 2627.687425] Lustre: 123025:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2627.708338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 2627.708338] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 2628.984649] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2629.472628] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2632.905922] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 21:05:28 (1768010728) [ 2634.266621] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2636.719424] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2637.281670] Lustre: Failing over lustre-MDT0000 [ 2637.394036] Lustre: server umount lustre-MDT0000 complete [ 2651.292202] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2651.293794] LDISKFS-fs (dm-0): recovery complete [ 2651.296859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2652.749901] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2656.758527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3137) [ 2656.758518] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 2657.917924] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2658.490511] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2661.724243] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 21:05:57 (1768010757) [ 2662.058061] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2665.489215] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2666.061740] Lustre: Failing over lustre-MDT0000 [ 2666.139343] Lustre: server umount lustre-MDT0000 complete [ 2680.216029] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2680.217467] LDISKFS-fs (dm-0): recovery complete [ 2680.220848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.713567] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2685.429759] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 2685.429789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3169) [ 2686.500500] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2687.074030] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2690.390449] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 21:06:25 (1768010785) [ 2690.730875] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2694.464277] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2695.116653] Lustre: Failing over lustre-MDT0000 [ 2695.208168] Lustre: server umount lustre-MDT0000 complete [ 2709.148520] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2709.150484] LDISKFS-fs (dm-0): recovery complete [ 2709.153736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.533836] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2714.616735] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3169) [ 2714.616765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201) [ 2718.943264] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 21:06:54 (1768010814) [ 2720.306342] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2720.307991] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2720.310328] LustreError: 16480:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc14633dc00 x1853890942964096/t257698037777(0) o35->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:147/0 lens 392/456 e 0 to 0 dl 1768010827 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2721.239626] Lustre: Failing over lustre-MDT0000 [ 2721.404423] Lustre: server umount lustre-MDT0000 complete [ 2734.181168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2735.596470] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2739.700214] Lustre: 16480:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc281b13b80 x1853890942964096/t257698037777(0) o35->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:166/0 lens 392/456 e 0 to 0 dl 1768010846 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2739.708493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 2739.712499] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3201) [ 2740.925427] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2741.510344] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2744.964996] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 21:07:20 (1768010840) [ 2745.326250] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2745.328254] LustreError: 6490:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc25caba680 x1853890942976768/t261993005072(0) o36->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:216/0 lens 504/448 e 0 to 0 dl 1768010896 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2748.927825] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2749.548911] Lustre: Failing over lustre-MDT0000 [ 2749.639680] Lustre: server umount lustre-MDT0000 complete [ 2763.590742] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2763.592391] LDISKFS-fs (dm-0): recovery complete [ 2763.595715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.040313] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2768.880943] Lustre: 113659:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc25fb2df80 x1853890942976768/t261993005072(0) o36->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:239/0 lens 504/2880 e 0 to 0 dl 1768010919 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2768.888536] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 2768.888825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3265) [ 2770.076477] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2770.612544] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2774.110886] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 21:07:49 (1768010869) [ 2774.482818] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2774.484429] LustreError: 6489:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc25c7d1f80 x1853890942990720/t266287972368(0) o36->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:245/0 lens 504/448 e 0 to 0 dl 1768010925 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2775.770249] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2778.197933] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2778.840873] Lustre: Failing over lustre-MDT0000 [ 2778.931291] Lustre: server umount lustre-MDT0000 complete [ 2793.045141] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2793.047220] LDISKFS-fs (dm-0): recovery complete [ 2793.049737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.472745] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2798.575290] Lustre: 113659:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc257e25500 x1853890942990720/t266287972368(0) o36->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:269/0 lens 504/2880 e 0 to 0 dl 1768010949 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2798.582986] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3265) [ 2798.583025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3297) [ 2798.584486] Lustre: 113659:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 2803.000453] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 21:08:18 (1768010898) [ 2803.346502] LustreError: 15174:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc2796daa00 x1853890943003520/t270582939664(0) o36->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:274/0 lens 504/448 e 0 to 0 dl 1768010954 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2803.356677] LustreError: 15174:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 2804.630621] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2804.632647] Lustre: Skipped 1 previous similar message [ 2807.315289] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2807.917831] Lustre: Failing over lustre-MDT0000 [ 2808.001305] Lustre: server umount lustre-MDT0000 complete [ 2822.072951] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2822.074955] LDISKFS-fs (dm-0): recovery complete [ 2822.077531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.503541] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2827.247866] Lustre: 6489:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc14611e680 x1853890943003520/t270582939664(0) o36->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:298/0 lens 504/2880 e 0 to 0 dl 1768010978 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2827.254803] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 2827.256497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3329) [ 2831.451867] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 21:08:47 (1768010927) [ 2831.820767] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2833.115506] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2833.117049] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2833.118357] Lustre: Skipped 2 previous similar messages [ 2836.522899] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2837.165633] Lustre: Failing over lustre-MDT0000 [ 2837.257479] Lustre: server umount lustre-MDT0000 complete [ 2851.369957] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2851.371078] LDISKFS-fs (dm-0): recovery complete [ 2851.373974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.823982] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2856.945381] Lustre: 16480:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc147401f80 x1853890943015552/t274877906960(0) o35->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:284/0 lens 392/456 e 0 to 0 dl 1768010964 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2856.951933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329) [ 2856.951978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3361) [ 2861.260246] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 21:09:16 (1768010956) [ 2861.561719] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 2861.564110] LustreError: 6491:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc278e4ea00 x1853890943025920/t279172874255(0) o101->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:332/0 lens 664/608 e 0 to 0 dl 1768011012 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2861.572174] LustreError: 6491:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 2921.575588] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnecting [ 2921.579129] Lustre: Skipped 4 previous similar messages [ 2921.585805] Lustre: 15174:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc145cc3b80 x1853890943025920/t279172874255(0) o101->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:392/0 lens 664/3488 e 0 to 0 dl 1768011072 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2923.838559] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 21:10:19 (1768011019) [ 2926.208584] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2926.811695] Lustre: Failing over lustre-MDT0000 [ 2926.877234] LustreError: 138960:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2926.878951] LustreError: 138960:0:(obd_class.h:479:obd_check_dev()) Skipped 107 previous similar messages [ 2926.902590] Lustre: server umount lustre-MDT0000 complete [ 2928.608474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.609221] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2928.612347] Lustre: Skipped 56 previous similar messages [ 2928.617847] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 179 previous similar messages [ 2940.812073] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2940.813697] LDISKFS-fs (dm-0): recovery complete [ 2940.817434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.855316] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2940.858468] LustreError: Skipped 11 previous similar messages [ 2940.929734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.932544] Lustre: Skipped 13 previous similar messages [ 2942.049779] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2942.054093] Lustre: Skipped 12 previous similar messages [ 2942.215107] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2946.018288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2946.020221] Lustre: Skipped 54 previous similar messages [ 2946.028527] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2946.031163] Lustre: Skipped 12 previous similar messages [ 2946.048524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3393) [ 2946.048536] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 2947.170295] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2947.707420] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2961.279599] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 21:10:56 (1768011056) [ 2964.105487] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2964.739885] Lustre: Failing over lustre-MDT0000 [ 2964.931648] Lustre: server umount lustre-MDT0000 complete [ 2966.497335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2966.499906] LustreError: Skipped 13 previous similar messages [ 2978.913523] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2978.915071] LDISKFS-fs (dm-0): recovery complete [ 2978.917374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2980.208318] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2984.449601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 2984.449614] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3393) [ 2985.563516] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2986.082533] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2988.333780] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2993.469162] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 21:11:29 (1768011089) [ 3002.377335] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3002.980918] Lustre: Failing over lustre-MDT0000 [ 3003.247976] Lustre: server umount lustre-MDT0000 complete [ 3017.259822] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3017.261160] LDISKFS-fs (dm-0): recovery complete [ 3017.263432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.769819] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3022.896149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 3022.896184] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 3024.047859] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3024.587106] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3037.679927] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 21:12:13 (1768011133) [ 3041.299399] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3041.937069] Lustre: Failing over lustre-MDT0000 [ 3042.059646] Lustre: server umount lustre-MDT0000 complete [ 3056.061970] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3056.063334] LDISKFS-fs (dm-0): recovery complete [ 3056.065810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.496570] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3061.247459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 3061.247477] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 3062.319041] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3062.824335] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3065.523129] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 3066.067526] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3068.181891] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 21:12:43 (1768011163) [ 3073.998769] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3135.410014] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3135.411544] Lustre: Skipped 1 previous similar message [ 3135.412749] LustreError: 113659:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc25572ce00 x1853890945812736/t296352743435(0) o36->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:606/0 lens 66040/440 e 0 to 0 dl 1768011286 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3194.979571] Lustre: 6489:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc2805f6850 x1853890945812736/t296352743435(0) o36->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:666/0 lens 66040/440 e 0 to 0 dl 1768011346 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3197.751132] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3198.266379] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 21:14:53 (1768011293) [ 3202.356874] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3203.171954] Lustre: Failing over lustre-MDT0000 [ 3203.430223] Lustre: server umount lustre-MDT0000 complete [ 3217.613353] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3217.615257] LDISKFS-fs (dm-0): recovery complete [ 3217.618447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3217.745161] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3217.748749] Lustre: Skipped 13 previous similar messages [ 3218.974387] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3223.168272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865) [ 3223.168276] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833) [ 3224.302846] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.802770] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3228.442422] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 21:15:24 (1768011324) [ 3233.699799] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3236.625430] Lustre: Failing over lustre-OST0000 [ 3236.666913] Lustre: server umount lustre-OST0000 complete [ 3250.672636] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3250.673747] LDISKFS-fs (dm-2): recovery complete [ 3250.676730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3252.570857] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3264.021897] Lustre: Failing over lustre-OST0000 [ 3264.066331] Lustre: server umount lustre-OST0000 complete [ 3276.798372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3278.695387] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3281.672116] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3282.200484] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3316.023139] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 21:16:51 (1768011411) [ 3316.927296] Lustre: Failing over lustre-MDT0000 [ 3317.120070] Lustre: server umount lustre-MDT0000 complete [ 3329.867664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3331.258217] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3335.159745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897) [ 3335.159777] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865) [ 3342.677876] Lustre: Failing over lustre-MDT0000 [ 3342.860792] Lustre: server umount lustre-MDT0000 complete [ 3355.591135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.048231] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3360.757105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929) [ 3360.757130] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897) [ 3361.870936] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3362.387739] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3365.727498] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 21:17:41 (1768011461) [ 3376.857253] Lustre: Failing over lustre-OST0000 [ 3376.895238] Lustre: server umount lustre-OST0000 complete [ 3389.614209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3391.492971] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3394.504490] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3395.000701] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3398.650900] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 21:18:14 (1768011494) [ 3399.260903] Lustre: Failing over lustre-MDT0000 [ 3399.524421] Lustre: server umount lustre-MDT0000 complete [ 3402.654084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3402.684488] Lustre: *** cfs_fail_loc=605, val=0*** [ 3402.685911] LustreError: 156712:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0ac0dd0 failed: rc = -95 [ 3402.689447] LustreError: 156712:0:(obd_config.c:783:class_setup()) setup MGS failed (-95) [ 3402.691895] LustreError: 156712:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 3402.694557] LustreError: 156712:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 3402.697212] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3402.699623] LustreError: 156712:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3402.704159] Lustre: server umount lustre-MDT0000 complete [ 3402.705829] LustreError: 156712:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3405.239478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3406.572805] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3409.151712] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 21:18:24 (1768011504) [ 3410.421046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 3410.421052] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 3412.837768] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3413.806125] Lustre: Failing over lustre-MDT0000 [ 3414.072031] Lustre: server umount lustre-MDT0000 complete [ 3427.947303] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3427.948703] LDISKFS-fs (dm-0): recovery complete [ 3427.952080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3429.389499] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3430.499124] Lustre: *** cfs_fail_loc=707, val=0*** [ 3490.923403] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 3490.934353] Lustre: 158865:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3490.937284] Lustre: 158865:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 3491.025954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 3491.025966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 3492.293205] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3492.803585] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3496.347494] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 21:19:51 (1768011591) [ 3519.709470] LustreError: 113659:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9fc281b3f100 x1853890946689792/t0(0) o101->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:191/0 lens 664/0 e 0 to 0 dl 1768011626 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3519.715088] LustreError: 113659:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3530.744075] LustreError: 113659:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3530.749132] LustreError: 16480:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9fc281b73480 x1853890946690816/t0(0) o35->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:202/0 lens 392/0 e 0 to 0 dl 1768011637 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3536.865944] LustreError: 42012:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9fc25c424e00 x1853890952189312/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:208/0 lens 544/0 e 0 to 0 dl 1768011643 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 3536.874970] LustreError: 42012:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 48 previous similar messages [ 3542.338521] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 21:20:37 (1768011637) [ 3565.495526] LustreError: 40973:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 3576.520062] LustreError: 40973:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 3579.284477] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 21:21:14 (1768011674) [ 3601.973721] LustreError: 24333:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9fc25c427100 x1853890946712832/t0(0) o101->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:274/0 lens 576/0 e 0 to 0 dl 1768011709 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3601.979960] LustreError: 24333:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3607.080056] LustreError: 24333:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3607.084169] LustreError: 24333:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9fc25cb4ce00 x1853890946713600/t0(0) o101->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:279/0 lens 664/0 e 0 to 0 dl 1768011714 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3617.832055] LustreError: 6489:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3617.836585] LustreError: 6489:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9fc257f06680 x1853890946735104/t0(0) o101->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:329/0 lens 664/0 e 0 to 0 dl 1768011764 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3617.841950] LustreError: 6489:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 3629.384179] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 21:22:05 (1768011725) [ 3714.023758] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 21:23:29 (1768011809) [ 3736.684864] LustreError: 6490:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9fc2796b7100 x1853890946788736/t0(0) o101->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:408/0 lens 576/0 e 0 to 0 dl 1768011843 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3736.691270] LustreError: 6490:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 3736.693698] LustreError: 6490:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3736.696174] LustreError: 6490:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 3737.112062] LustreError: 6490:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3752.921648] LustreError: 6489:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3752.924478] LustreError: 6489:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3753.344051] LustreError: 6489:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3753.346996] LustreError: 6489:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3768.753831] LustreError: 6489:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9fc2819ce680 x1853890946807936/t0(0) o36->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:440/0 lens 504/0 e 0 to 0 dl 1768011875 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 3768.762520] LustreError: 6489:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 82 previous similar messages [ 3780.796222] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 21:24:36 (1768011876) [ 3805.034441] Lustre: DEBUG MARKER: phase 2 [ 3807.978380] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 21:25:03 (1768011903) [ 3877.925830] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 21:26:13 (1768011973) [ 3878.346486] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 3878.829760] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 21:26:14 (1768011974) [ 3880.484670] Lustre: DEBUG MARKER: Started rundbench load pid=133573 ... [ 3884.012823] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3885.500120] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 3886.161955] Lustre: Failing over lustre-MDT0000 [ 3886.162435] Lustre: 24333:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff9fc142b59500 x1853890952494720/t0(0) o104->lustre-MDT0000@192.168.204.31@tcp:15/16 lens 328/224 e 0 to 1 dl 0 ref 1 fl Rpc:EeXQU/0/ffffffff rc -19/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 [ 3886.169859] LustreError: 24333:0:(ldlm_lockd.c:1452:ldlm_handle_enqueue()) ### lock on destroyed export 00000000a597441c ns: mdt-lustre-MDT0000_UUID lock: ffff9fc14cd6b400/0x88e2a32918ca79b8 lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0xc43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.204.31@tcp remote: 0xc3c0ae35a2ce3232 expref: 3 pid: 24333 timeout: 0 lvb_type: 0 [ 3886.190420] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 3886.396457] LustreError: 165755:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3886.398113] LustreError: 165755:0:(obd_class.h:479:obd_check_dev()) Skipped 83 previous similar messages [ 3886.428020] Lustre: server umount lustre-MDT0000 complete [ 3889.121133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3889.121770] LustreError: 113659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3889.125823] Lustre: Skipped 42 previous similar messages [ 3889.129897] LustreError: 113659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 156 previous similar messages [ 3900.064250] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3900.065511] LDISKFS-fs (dm-0): recovery complete [ 3900.068725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.103725] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3900.106553] LustreError: Skipped 8 previous similar messages [ 3900.180652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.182460] Lustre: Skipped 11 previous similar messages [ 3900.192552] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3900.194382] Lustre: Skipped 7 previous similar messages [ 3901.347494] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3901.537786] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3901.540054] Lustre: Skipped 11 previous similar messages [ 3905.512640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3905.514228] Lustre: Skipped 41 previous similar messages [ 3905.918330] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3905.920363] Lustre: Skipped 11 previous similar messages [ 3905.935770] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5100 to 0x2c0000401:5121) [ 3905.935826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5136 to 0x280000401:5153) [ 3907.048367] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3907.561169] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3913.137541] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3914.801230] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 3915.673259] Lustre: Failing over lustre-MDT0001 [ 3915.745166] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3915.746165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3915.747297] LustreError: Skipped 7 previous similar messages [ 3915.852762] Lustre: server umount lustre-MDT0001 complete [ 3930.427324] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 3930.429554] LDISKFS-fs (dm-1): recovery complete [ 3930.434747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3931.875462] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3935.750121] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:421 to 0x280000400:449) [ 3935.750196] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:449) [ 3937.091812] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3937.710546] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3943.686811] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3945.411593] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 3946.146856] Lustre: Failing over lustre-MDT0000 [ 3946.153255] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 3946.157555] Lustre: Skipped 2 previous similar messages [ 3946.293389] Lustre: server umount lustre-MDT0000 complete [ 3962.075453] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3962.077703] LDISKFS-fs (dm-0): recovery complete [ 3962.081565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.129471] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3968.050522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5454 to 0x280000401:5473) [ 3968.050528] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5421 to 0x2c0000401:5441) [ 3969.361724] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3969.964122] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3975.574544] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3977.402602] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 3978.288743] Lustre: Failing over lustre-MDT0001 [ 3978.338113] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 3978.502971] Lustre: server umount lustre-MDT0001 complete [ 3994.366842] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 3994.370800] LDISKFS-fs (dm-1): recovery complete [ 3994.381621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3996.269886] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3999.762835] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:481) [ 3999.762983] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:421 to 0x280000400:481) [ 4001.241187] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4001.936633] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4027.176703] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 21:28:42 (1768012122) [ 4150.651796] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4161.488453] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 4162.142444] Lustre: Failing over lustre-MDT0001 [ 4162.146658] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 4168.268980] Lustre: server umount lustre-MDT0001 complete [ 4181.968281] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4181.969359] LDISKFS-fs (dm-1): recovery complete [ 4181.973035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4183.187371] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4188.784651] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1467 to 0x280000400:1505) [ 4188.784699] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1467 to 0x2c0000400:1505) [ 4189.861505] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4190.366124] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4315.278298] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4326.017875] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 4326.769196] Lustre: Failing over lustre-MDT0000 [ 4326.769346] Lustre: 113659:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff9fc142b58000 x1853890962145024/t0(0) o104->lustre-MDT0000@192.168.204.31@tcp:15/16 lens 328/224 e 0 to 1 dl 0 ref 1 fl Rpc:EeXQU/0/ffffffff rc -19/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 [ 4326.785622] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 4326.787722] Lustre: Skipped 4 previous similar messages [ 4333.213379] Lustre: server umount lustre-MDT0000 complete [ 4348.040371] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4348.042840] LDISKFS-fs (dm-0): recovery complete [ 4348.047641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.679753] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4355.649627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7849 to 0x280000401:7873) [ 4355.649710] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7816 to 0x2c0000401:7841) [ 4356.861793] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4357.439405] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4376.597055] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 21:34:32 (1768012472) [ 4499.497112] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4510.246427] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 4510.844915] Lustre: Failing over lustre-MDT0000 [ 4510.867599] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 4510.871695] Lustre: Skipped 5 previous similar messages [ 4512.224798] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.228111] Lustre: Skipped 19 previous similar messages [ 4516.899336] LustreError: 176375:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4516.901168] LustreError: 176375:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 4516.945428] Lustre: server umount lustre-MDT0000 complete [ 4517.344537] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.349073] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 77 previous similar messages [ 4531.067604] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4531.068952] LDISKFS-fs (dm-0): recovery complete [ 4531.071649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.108696] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4531.111220] LustreError: Skipped 2 previous similar messages [ 4531.174497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4531.176841] Lustre: Skipped 5 previous similar messages [ 4531.187078] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4531.188860] Lustre: Skipped 5 previous similar messages [ 4531.297936] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4531.300284] Lustre: Skipped 5 previous similar messages [ 4532.465212] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4536.292108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4536.295946] Lustre: Skipped 20 previous similar messages [ 4538.019977] Lustre: lustre-MDT0000: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 4538.025447] Lustre: Skipped 5 previous similar messages [ 4538.045242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8388 to 0x280000401:8417) [ 4538.045678] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8356 to 0x2c0000401:8385) [ 4539.418184] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4539.966305] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4663.965049] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4674.892475] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 4675.611952] Lustre: Failing over lustre-MDT0000 [ 4675.640980] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 4675.642828] Lustre: Skipped 5 previous similar messages [ 4676.576781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4676.581253] LustreError: Skipped 4 previous similar messages [ 4681.288162] Lustre: server umount lustre-MDT0000 complete [ 4695.200640] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4695.201920] LDISKFS-fs (dm-0): recovery complete [ 4695.205691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.434966] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4701.569116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8781 to 0x280000401:8801) [ 4701.569131] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8750 to 0x2c0000401:8769) [ 4703.056370] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4703.812530] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4707.969779] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 21:40:03 (1768012803) [ 4830.925208] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4841.637643] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 4842.198070] Lustre: Failing over lustre-MDT0001 [ 4842.201542] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4842.203143] Lustre: Skipped 6 previous similar messages [ 4842.405645] Lustre: server umount lustre-MDT0001 complete [ 4856.047243] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4856.048426] LDISKFS-fs (dm-1): recovery complete [ 4856.051640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4857.233482] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4861.454830] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3295 to 0x280000400:3329) [ 4861.454847] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3294 to 0x2c0000400:3329) [ 4862.871172] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4863.587860] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4987.461155] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4998.223782] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 4998.850704] Lustre: Failing over lustre-MDT0001 [ 4998.863513] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4999.152142] Lustre: server umount lustre-MDT0001 complete [ 5012.928370] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5012.929700] LDISKFS-fs (dm-1): recovery complete [ 5012.933036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5014.235480] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5018.105243] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3294 to 0x2c0000400:3361) [ 5018.105260] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3295 to 0x280000400:3361) [ 5019.371729] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5020.054982] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5023.893717] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 21:45:19 (1768013119) [ 5029.483752] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5030.970155] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 5031.550707] Lustre: Failing over lustre-OST0000 [ 5031.590579] Lustre: server umount lustre-OST0000 complete [ 5047.014191] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5047.017245] LDISKFS-fs (dm-2): recovery complete [ 5047.024385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5049.690223] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5053.498247] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5054.191918] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5063.632589] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5065.116610] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5065.699110] Lustre: Failing over lustre-OST0000 [ 5065.731146] Lustre: server umount lustre-OST0000 complete [ 5079.574330] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5079.575704] LDISKFS-fs (dm-2): recovery complete [ 5079.578475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5081.404662] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5084.697959] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5085.399392] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5091.760059] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 21:46:27 (1768013187) [ 5215.883081] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5219.175754] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5230.039800] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 5230.850739] Lustre: Failing over lustre-MDT0000 [ 5231.002201] LustreError: 187771:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5231.005248] LustreError: 187771:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 5231.064412] Lustre: server umount lustre-MDT0000 complete [ 5232.456664] LustreError: 13157:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768013328 with bad export cookie 9863625530803991552 [ 5232.457883] Lustre: Failing over lustre-MDT0001 [ 5232.458300] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5232.458306] LustreError: Skipped 1 previous similar message [ 5232.460330] LustreError: 13157:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5232.585691] Lustre: server umount lustre-MDT0001 complete [ 5247.804813] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5247.807212] LDISKFS-fs (dm-1): recovery complete [ 5247.812093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5247.858279] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5247.859542] LDISKFS-fs (dm-0): recovery complete [ 5247.862575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.897856] LustreError: 189068:0:(llog.c:1643:llog_backup()) MGC192.168.204.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5247.900818] Lustre: 189068:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5247.968558] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5247.968934] LustreError: 189078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5247.975185] Lustre: Skipped 18 previous similar messages [ 5247.984305] LustreError: 189078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 69 previous similar messages [ 5248.992068] Lustre: 3627:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768013329/real 1768013329] req@ffff9fc14f4c2a00 x1853890987415552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768013345 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5248.998369] Lustre: 3627:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5253.088242] Lustre: 3630:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768013333/real 1768013333] req@ffff9fc1471d3800 x1853890987415936/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768013349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5253.102229] Lustre: 3630:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5257.184655] LustreError: 3626:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc145956680 x1853890987419008/t0(0) o250->MGC192.168.204.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5257.303081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5257.306633] Lustre: Skipped 5 previous similar messages [ 5257.328666] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5257.333690] Lustre: Skipped 5 previous similar messages [ 5258.336091] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768013338/real 1768013338] req@ffff9fc1471d2a00 x1853890987416704/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768013354 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5258.545193] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5258.620354] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5258.806086] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5258.808956] Lustre: Skipped 6 previous similar messages [ 5262.821196] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5262.825070] Lustre: Skipped 17 previous similar messages [ 5263.468074] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5263.474032] Lustre: Skipped 5 previous similar messages [ 5263.501935] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3294 to 0x2c0000400:3393) [ 5263.501963] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3295 to 0x280000400:3393) [ 5268.953921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12210 to 0x280000401:12225) [ 5268.954820] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12179 to 0x2c0000401:12225) [ 5270.432310] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5271.140349] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5271.776634] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5276.041187] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 21:49:31 (1768013371) [ 5279.627371] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5280.925107] Lustre: Failing over lustre-MDT0000 [ 5281.105774] Lustre: server umount lustre-MDT0000 complete [ 5284.321111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5284.325890] LustreError: Skipped 4 previous similar messages [ 5296.246940] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5296.250409] LDISKFS-fs (dm-0): recovery complete [ 5296.257534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.958895] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5299.300754] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5315.703639] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5315.765112] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12179 to 0x2c0000401:12257) [ 5315.765123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12257) [ 5317.625369] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5318.377438] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5322.280441] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 21:50:17 (1768013417) [ 5325.478533] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5326.382852] Lustre: Failing over lustre-MDT0000 [ 5326.558616] Lustre: server umount lustre-MDT0000 complete [ 5342.130078] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5342.133386] LDISKFS-fs (dm-0): recovery complete [ 5342.139244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5344.020209] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5345.392283] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5345.395793] LustreError: 189098:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc14876f850 x1853891019811328/t339302451280(339302451280) o101->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:507/0 lens 520/664 e 0 to 0 dl 1768013452 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5361.802364] Lustre: lustre-MDT0000: Client a9792733-3f58-4d18-919d-c5231f8fa15b (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5361.811626] Lustre: 189831:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc2790d9f80 x1853891019811328/t339302451280(339302451280) o101->a9792733-3f58-4d18-919d-c5231f8fa15b@192.168.204.31@tcp:523/0 lens 520/3488 e 0 to 0 dl 1768013468 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5361.871301] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12259 to 0x2c0000401:12289) [ 5361.871334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12289) [ 5363.611753] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5364.408700] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5368.556234] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 21:51:04 (1768013464) [ 5369.873682] Lustre: Failing over lustre-OST0000 [ 5369.919373] Lustre: server umount lustre-OST0000 complete [ 5371.245545] Lustre: Failing over lustre-MDT0000 [ 5371.331960] Lustre: server umount lustre-MDT0000 complete [ 5385.115518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.154389] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5388.281539] Lustre: lustre-MDT0000: Denying connection for new client 59ccc435-3e28-4d07-84e3-d520f2a1ba8a (at 192.168.204.31@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5390.344946] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12259 to 0x2c0000401:12321) [ 5396.242740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5398.092741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12321) [ 5398.990822] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5403.922070] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 21:51:39 (1768013499) [ 5404.453218] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5404.456275] LustreError: 175568:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc279521880 x1853890988631808/t356482285575(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:566/0 lens 1312/4320 e 0 to 0 dl 1768013511 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5408.074838] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5408.998139] Lustre: Failing over lustre-MDT0000 [ 5409.220085] Lustre: server umount lustre-MDT0000 complete [ 5420.000180] Lustre: 189106:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768013500/real 1768013500] req@ffff9fc279523480 x1853890988631808/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1768013516 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5420.014323] Lustre: 189106:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5424.242308] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5424.244888] LDISKFS-fs (dm-0): recovery complete [ 5424.250635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5426.048092] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5429.748056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12353) [ 5429.748136] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12323 to 0x2c0000401:12353) [ 5431.245507] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5432.113742] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5436.616501] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 21:52:11 (1768013531) [ 5437.100816] LustreError: 8383:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc2644f0a80 x1853890988660480/t360777252876(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:599/0 lens 2520/4320 e 0 to 0 dl 1768013544 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5440.163003] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5443.544066] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5444.392370] Lustre: Failing over lustre-MDT0001 [ 5444.399725] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 5444.403841] Lustre: Skipped 1 previous similar message [ 5450.356593] Lustre: server umount lustre-MDT0001 complete [ 5465.774953] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5465.777760] LDISKFS-fs (dm-1): recovery complete [ 5465.786243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5467.565114] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5471.229658] Lustre: 189102:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc25c7df800 x1853891024917888/t30064773038(0) o36->59ccc435-3e28-4d07-84e3-d520f2a1ba8a@192.168.204.31@tcp:658/0 lens 560/2880 e 0 to 0 dl 1768013603 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5471.241782] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3404 to 0x2c0000400:3425) [ 5471.241933] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3404 to 0x280000400:3425) [ 5472.798899] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5473.543881] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5478.088347] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 21:52:53 (1768013573) [ 5478.624162] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5478.625964] Lustre: Skipped 1 previous similar message [ 5481.986567] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5485.532481] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5486.465596] Lustre: Failing over lustre-MDT0000 [ 5486.756141] Lustre: server umount lustre-MDT0000 complete [ 5494.752255] Lustre: 199454:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768013574/real 1768013574] req@ffff9fc25d2b8380 x1853890988695168/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1768013590 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5500.821218] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5500.822757] LDISKFS-fs (dm-0): recovery complete [ 5500.825981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.131263] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5506.070737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12385) [ 5506.070788] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12323 to 0x2c0000401:12385) [ 5507.364979] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5508.174877] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5510.292604] Lustre: Failing over lustre-MDT0001 [ 5510.394392] Lustre: server umount lustre-MDT0001 complete [ 5524.773798] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5524.775539] LDISKFS-fs (dm-1): recovery complete [ 5524.781517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5526.501407] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5530.118799] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3436 to 0x2c0000400:3457) [ 5530.118859] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3436 to 0x280000400:3457) [ 5531.831399] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5532.506893] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5537.477444] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 21:53:52 (1768013632) [ 5537.983770] LustreError: 8384:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc148602680 x1853890988744832/t365072220176(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:700/0 lens 2520/4320 e 0 to 0 dl 1768013645 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5537.996536] LustreError: 8384:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 5544.071919] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5547.489579] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5548.183109] Lustre: Failing over lustre-MDT0000 [ 5548.452465] Lustre: server umount lustre-MDT0000 complete [ 5549.669142] LustreError: 7866:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768013645 with bad export cookie 9863625530806520540 [ 5549.672111] Lustre: Failing over lustre-MDT0001 [ 5549.675966] LustreError: 7866:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5555.169513] LustreError: 204758:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9fc148b2ed80 x1853890988756352/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5555.181931] LustreError: 204758:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5555.327973] Lustre: server umount lustre-MDT0001 complete [ 5571.099569] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5571.101907] LDISKFS-fs (dm-0): recovery complete [ 5571.103093] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5571.105820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5571.105923] LDISKFS-fs (dm-1): recovery complete [ 5571.113576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5571.196111] LustreError: 205857:0:(llog.c:1643:llog_backup()) MGC192.168.204.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5571.199474] Lustre: 205857:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5574.112359] LustreError: 3626:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc25fbcd880 x1853890988757504/t0(0) o250->MGC192.168.204.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5575.563638] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5575.687539] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5580.350212] Lustre: 205906:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc25fbce680 x1853891024988160/t38654705734(0) o36->59ccc435-3e28-4d07-84e3-d520f2a1ba8a@192.168.204.31@tcp:12/0 lens 560/2880 e 0 to 0 dl 1768013712 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5580.355980] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3468 to 0x2c0000400:3489) [ 5580.355991] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3468 to 0x280000400:3489) [ 5580.389681] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12323 to 0x2c0000401:12417) [ 5580.389685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12417) [ 5581.782736] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5582.548728] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5583.163572] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5587.750623] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 21:54:43 (1768013683) [ 5594.240840] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5595.048876] Lustre: Failing over lustre-MDT0000 [ 5595.353932] Lustre: server umount lustre-MDT0000 complete [ 5604.451079] Lustre: lustre-MDT0001: Client 59ccc435-3e28-4d07-84e3-d520f2a1ba8a (at 192.168.204.31@tcp) reconnecting [ 5604.457987] Lustre: Skipped 2 previous similar messages [ 5604.465417] Lustre: 206814:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc144698700 x1853891025021696/t42949673030(0) o36->59ccc435-3e28-4d07-84e3-d520f2a1ba8a@192.168.204.31@tcp:11/0 lens 560/2880 e 0 to 0 dl 1768013711 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5610.646246] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5610.648862] LDISKFS-fs (dm-0): recovery complete [ 5610.654173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5612.624399] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5616.148939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12449) [ 5616.148989] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12323 to 0x2c0000401:12449) [ 5617.612831] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5618.357935] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5623.163435] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 21:55:18 (1768013718) [ 5623.594093] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5623.595625] Lustre: Skipped 2 previous similar messages [ 5627.054121] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5627.944786] Lustre: Failing over lustre-MDT0001 [ 5628.067734] Lustre: server umount lustre-MDT0001 complete [ 5643.664966] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5643.668982] LDISKFS-fs (dm-1): recovery complete [ 5643.677964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5645.506637] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5648.885084] Lustre: 206814:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc257c32d80 x1853891025053568/t42949673101(0) o36->59ccc435-3e28-4d07-84e3-d520f2a1ba8a@192.168.204.31@tcp:55/0 lens 560/2880 e 0 to 0 dl 1768013755 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5648.893727] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3510 to 0x2c0000400:3553) [ 5648.893756] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3510 to 0x280000400:3553) [ 5650.480916] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5651.224872] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5655.901452] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 21:55:51 (1768013751) [ 5662.941450] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5666.405672] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5667.279745] Lustre: Failing over lustre-MDT0000 [ 5667.540811] Lustre: server umount lustre-MDT0000 complete [ 5672.547097] Lustre: lustre-MDT0001: Client 59ccc435-3e28-4d07-84e3-d520f2a1ba8a (at 192.168.204.31@tcp) reconnecting [ 5683.260177] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5683.263064] LDISKFS-fs (dm-0): recovery complete [ 5683.271673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.277545] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5688.860093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12481) [ 5688.860229] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12323 to 0x2c0000401:12481) [ 5690.467239] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5691.146740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5693.485296] Lustre: Failing over lustre-MDT0001 [ 5693.622485] Lustre: server umount lustre-MDT0001 complete [ 5708.827922] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5708.830263] LDISKFS-fs (dm-1): recovery complete [ 5708.837178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5710.543017] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5714.436342] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3564 to 0x280000400:3585) [ 5714.436347] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3564 to 0x2c0000400:3585) [ 5716.086790] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5716.825401] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5721.427199] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 21:56:56 (1768013816) [ 5721.888429] LustreError: 205906:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc2790da300 x1853891025122176/t51539607622(0) o36->59ccc435-3e28-4d07-84e3-d520f2a1ba8a@192.168.204.31@tcp:128/0 lens 560/448 e 0 to 0 dl 1768013828 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5721.900204] LustreError: 205906:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 3 previous similar messages [ 5728.213645] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5731.802087] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5732.789707] Lustre: Failing over lustre-MDT0000 [ 5733.045794] Lustre: server umount lustre-MDT0000 complete [ 5734.756991] LustreError: 6474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768013830 with bad export cookie 9863625530806534260 [ 5734.758513] Lustre: Failing over lustre-MDT0001 [ 5734.763844] LustreError: 6474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5734.901582] Lustre: server umount lustre-MDT0001 complete [ 5750.240317] Lustre: 3629:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768013830/real 1768013830] req@ffff9fc143361c00 x1853890988915584/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768013846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5750.250913] Lustre: 3629:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5751.202949] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5751.205633] LDISKFS-fs (dm-0): recovery complete [ 5751.209845] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5751.210873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5751.212471] LDISKFS-fs (dm-1): recovery complete [ 5751.221913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5759.456684] LustreError: 3626:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc14862b480 x1853890988918272/t0(0) o250->MGC192.168.204.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5761.334139] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5761.478872] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5762.680555] Lustre: 216745:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc148602680 x1853891025122176/t51539607622(0) o36->59ccc435-3e28-4d07-84e3-d520f2a1ba8a@192.168.204.31@tcp:169/0 lens 560/2880 e 0 to 0 dl 1768013869 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5762.693295] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3596 to 0x280000400:3617) [ 5762.693327] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3596 to 0x2c0000400:3617) [ 5762.697218] Lustre: 216745:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5762.747838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12513) [ 5762.747984] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12323 to 0x2c0000401:12513) [ 5765.176108] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5765.986652] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5766.717128] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5771.441305] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 21:57:46 (1768013866) [ 5775.619541] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5776.543813] Lustre: Failing over lustre-MDT0001 [ 5782.104206] Lustre: server umount lustre-MDT0001 complete [ 5797.733447] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5797.736230] LDISKFS-fs (dm-1): recovery complete [ 5797.743223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5799.592774] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5803.017908] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3628 to 0x280000400:3649) [ 5803.017923] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3628 to 0x2c0000400:3649) [ 5804.662731] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5805.522419] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5810.384696] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 21:58:25 (1768013905) [ 5814.641257] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5815.605473] Lustre: Failing over lustre-MDT0000 [ 5815.879396] Lustre: server umount lustre-MDT0000 complete [ 5827.552147] Lustre: 218746:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768013907/real 1768013907] req@ffff9fc25cf94380 x1853890988982656/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1984/4320 e 0 to 1 dl 1768013923 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5827.565604] Lustre: 218746:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 5831.950771] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5831.953560] LDISKFS-fs (dm-0): recovery complete [ 5831.958381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.964135] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5837.329405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12545) [ 5837.329444] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12323 to 0x2c0000401:12545) [ 5839.114767] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5839.968937] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5844.788798] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 21:59:00 (1768013940) [ 5848.920574] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5852.443118] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5853.340552] Lustre: Failing over lustre-MDT0000 [ 5853.588118] LustreError: 222241:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 5853.590818] LustreError: 222241:0:(obd_class.h:479:obd_check_dev()) Skipped 155 previous similar messages [ 5853.622885] Lustre: server umount lustre-MDT0000 complete [ 5854.832202] LustreError: 216455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5854.840813] LustreError: 216455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 208 previous similar messages [ 5857.761332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5857.771803] Lustre: Skipped 56 previous similar messages [ 5861.856175] Lustre: 218746:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768013941/real 1768013941] req@ffff9fc2790a9500 x1853890989010560/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1984/4320 e 0 to 1 dl 1768013957 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5869.269923] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5869.272709] LDISKFS-fs (dm-0): recovery complete [ 5869.278210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.334926] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5869.340975] LustreError: Skipped 10 previous similar messages [ 5869.443886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5869.447364] Lustre: Skipped 19 previous similar messages [ 5869.493425] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5869.495255] Lustre: Skipped 19 previous similar messages [ 5870.178498] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5870.184140] Lustre: Skipped 18 previous similar messages [ 5871.282554] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5874.661778] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5874.665552] Lustre: Skipped 65 previous similar messages [ 5874.687651] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5874.692566] Lustre: Skipped 19 previous similar messages [ 5874.721341] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12323 to 0x2c0000401:12577) [ 5874.721415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12577) [ 5876.369669] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5877.199911] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5879.596889] Lustre: Failing over lustre-MDT0001 [ 5879.733560] Lustre: server umount lustre-MDT0001 complete [ 5895.742441] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5895.745024] LDISKFS-fs (dm-1): recovery complete [ 5895.752473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5897.805771] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5901.319545] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3628 to 0x280000400:3681) [ 5901.319565] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3628 to 0x2c0000400:3681) [ 5902.922076] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5903.671206] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5907.632104] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 22:00:03 (1768014003) [ 5908.121878] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5908.125180] Lustre: Skipped 5 previous similar messages [ 5911.691515] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5915.250988] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5916.190333] Lustre: Failing over lustre-MDT0000 [ 5916.443692] Lustre: server umount lustre-MDT0000 complete [ 5916.641422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5916.646274] LustreError: Skipped 15 previous similar messages [ 5918.181140] LustreError: 144782:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768014014 with bad export cookie 9863625530806544683 [ 5918.182779] Lustre: Failing over lustre-MDT0001 [ 5918.188675] LustreError: 144782:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5918.192992] LustreError: 225921:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200028c71:0x1f:0x0].0xf7117594 (ffff9fc27968ef00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5923.810317] LustreError: 225921:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9fc25c861880 x1853890989065856/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5923.823458] LustreError: 225921:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5923.975609] Lustre: server umount lustre-MDT0001 complete [ 5940.054254] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5940.056681] LDISKFS-fs (dm-0): recovery complete [ 5940.058465] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5940.060631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.061139] LDISKFS-fs (dm-1): recovery complete [ 5940.069020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5943.264574] LustreError: 3626:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc14c50c380 x1853890989066624/t0(0) o250->MGC192.168.204.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5944.975346] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5945.046932] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5949.662740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12609) [ 5949.663426] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12323 to 0x2c0000401:12609) [ 5949.671175] Lustre: 227036:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fc14313f480 x1853891025201920/t64424509442(0) o36->59ccc435-3e28-4d07-84e3-d520f2a1ba8a@192.168.204.31@tcp:381/0 lens 496/2888 e 0 to 0 dl 1768014081 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 5949.690178] Lustre: 227036:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5949.697798] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3628 to 0x280000400:3713) [ 5949.697870] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3628 to 0x2c0000400:3713) [ 5951.401828] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5952.172862] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5952.830172] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5957.091396] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 22:00:52 (1768014052) [ 5961.629256] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5962.574733] Lustre: Failing over lustre-MDT0000 [ 5962.807919] Lustre: server umount lustre-MDT0000 complete [ 5973.110759] Lustre: lustre-MDT0001: Client 59ccc435-3e28-4d07-84e3-d520f2a1ba8a (at 192.168.204.31@tcp) reconnecting [ 5978.284694] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5978.287451] LDISKFS-fs (dm-0): recovery complete [ 5978.293406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5980.357972] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5983.770828] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12323 to 0x2c0000401:12641) [ 5983.770963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12641) [ 5985.303742] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5986.107964] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5990.607628] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 22:01:25 (1768014085) [ 5991.154763] LustreError: 227975:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9fc148601500 x1853891025229184/t68719476741(0) o36->59ccc435-3e28-4d07-84e3-d520f2a1ba8a@192.168.204.31@tcp:398/0 lens 496/456 e 0 to 0 dl 1768014098 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 5991.169551] LustreError: 227975:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 5 previous similar messages [ 5994.786785] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5995.744355] Lustre: Failing over lustre-MDT0001 [ 5995.867320] Lustre: server umount lustre-MDT0001 complete [ 6011.503931] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6011.506795] LDISKFS-fs (dm-1): recovery complete [ 6011.514520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6013.439359] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6017.023246] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3628 to 0x280000400:3745) [ 6017.023265] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3628 to 0x2c0000400:3745) [ 6018.597523] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6019.388278] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6023.908588] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 22:01:59 (1768014119) [ 6027.901945] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6031.577144] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6032.523001] Lustre: Failing over lustre-MDT0000 [ 6032.722534] Lustre: server umount lustre-MDT0000 complete [ 6048.639194] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6048.640899] LDISKFS-fs (dm-0): recovery complete [ 6048.643489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.414938] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6053.917703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12673) [ 6053.917721] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12323 to 0x2c0000401:12673) [ 6055.596759] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6056.360853] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6058.738827] Lustre: Failing over lustre-MDT0001 [ 6058.882448] Lustre: server umount lustre-MDT0001 complete [ 6074.909219] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6074.911876] LDISKFS-fs (dm-1): recovery complete [ 6074.919023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6076.792610] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6080.517398] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3628 to 0x280000400:3777) [ 6080.517434] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3628 to 0x2c0000400:3777) [ 6082.048589] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6082.878885] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6086.563505] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 22:03:02 (1768014182) [ 6090.126883] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6093.715199] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6094.638793] Lustre: Failing over lustre-MDT0000 [ 6094.835811] Lustre: server umount lustre-MDT0000 complete [ 6096.527381] LustreError: 6474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768014192 with bad export cookie 9863625530806552215 [ 6096.528923] Lustre: Failing over lustre-MDT0001 [ 6096.534209] LustreError: 6474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6096.660898] Lustre: server umount lustre-MDT0001 complete [ 6112.235200] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6112.238513] LDISKFS-fs (dm-0): recovery complete [ 6112.242992] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6112.244252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6112.246196] LDISKFS-fs (dm-1): recovery complete [ 6112.250854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6112.352334] LustreError: 237649:0:(llog.c:1643:llog_backup()) MGC192.168.204.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6112.355881] Lustre: 237649:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6117.344146] Lustre: 3630:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768014197/real 1768014197] req@ffff9fc264d8ea00 x1853890989202944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768014213 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6121.440993] LustreError: 3626:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc1410a1880 x1853890989205632/t0(0) o250->MGC192.168.204.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6123.257563] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6123.421992] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6127.750557] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3628 to 0x280000400:3809) [ 6127.750608] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3628 to 0x2c0000400:3809) [ 6127.783652] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12323 to 0x2c0000401:12705) [ 6127.783719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12227 to 0x280000401:12705) [ 6129.581481] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6130.388881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6131.167532] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6135.399151] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 22:03:50 (1768014230) [ 6136.196796] Lustre: 238888:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 59ccc435-3e28-4d07-84e3-d520f2a1ba8a at adminstrative request [ 6140.618943] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 22:03:55 (1768014235) [ 6143.469848] Lustre: Failing over lustre-MDT0000 [ 6143.648422] Lustre: server umount lustre-MDT0000 complete [ 6157.370405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6159.228597] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6162.998536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12757 to 0x280000401:12801) [ 6162.998555] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12756 to 0x2c0000401:12801) [ 6164.465986] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6165.199987] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6169.373778] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 22:04:24 (1768014264) [ 6175.241921] Lustre: Failing over lustre-OST0000 [ 6175.324689] Lustre: server umount lustre-OST0000 complete [ 6189.235370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6192.041921] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6196.056372] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6196.803587] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6201.357551] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 22:04:56 (1768014296) [ 6203.143119] Lustre: Failing over lustre-MDT0000 [ 6203.355135] Lustre: server umount lustre-MDT0000 complete [ 6206.938617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6208.927898] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6210.031318] Lustre: lustre-MDT0000: Denying connection for new client c3b5c144-cecf-4f60-89b2-54830e14a9e8 (at 192.168.204.31@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 6210.040983] Lustre: Skipped 2 previous similar messages [ 6212.616412] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12756 to 0x2c0000401:12833) [ 6212.616493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12757 to 0x280000401:12833) [ 6218.488104] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 22:05:13 (1768014313) [ 6222.463872] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6223.688612] Lustre: Failing over lustre-OST0000 [ 6223.741705] Lustre: server umount lustre-OST0000 complete [ 6239.300430] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6239.303057] LDISKFS-fs (dm-2): recovery complete [ 6239.308918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6241.751526] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6245.579158] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6246.169535] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6250.456959] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 22:05:45 (1768014345) [ 6253.787270] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6256.025369] Lustre: Failing over lustre-OST0000 [ 6256.073854] Lustre: server umount lustre-OST0000 complete [ 6271.094568] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6271.097470] LDISKFS-fs (dm-2): recovery complete [ 6271.103339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6272.600376] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.31@tcp inode [0x20002f9d1:0x5:0x0] object 0x280000401:12835 extent [0-4194303]: client csum e9f808fa, server csum a1b40a11 [ 6273.559869] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6277.093180] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6277.664748] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6282.004452] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 22:06:17 (1768014377) [ 6285.616479] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6289.079036] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6291.612901] Lustre: Failing over lustre-MDT0000 [ 6291.914570] Lustre: server umount lustre-MDT0000 complete [ 6293.614206] LustreError: 6475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768014389 with bad export cookie 9863625530806579004 [ 6293.617605] Lustre: Failing over lustre-OST0000 [ 6293.619447] LustreError: 6475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6293.677789] Lustre: server umount lustre-OST0000 complete [ 6309.241861] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6309.243917] LDISKFS-fs (dm-0): recovery complete [ 6309.248501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.562061] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x88e2a32919333e23 [ 6318.626256] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (not set up) [ 6318.630050] Lustre: Skipped 33 previous similar messages [ 6320.328834] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6323.913206] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12756 to 0x2c0000401:12865) [ 6336.596093] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6336.598744] LDISKFS-fs (dm-2): recovery complete [ 6336.604629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6339.223432] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6346.120285] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 22:07:21 (1768014441) [ 6351.313452] Lustre: Failing over lustre-OST0000 [ 6351.390664] Lustre: server umount lustre-OST0000 complete [ 6353.068232] Lustre: Failing over lustre-MDT0000 [ 6353.317349] Lustre: server umount lustre-MDT0000 complete [ 6366.561893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6368.275446] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6371.851435] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12756 to 0x2c0000401:12897) [ 6376.257734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6378.699674] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6379.753182] Lustre: lustre-OST0000: Denying connection for new client 251f9afe-be0a-4522-a288-c3e50cc32311 (at 192.168.204.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 6379.761661] Lustre: Skipped 1 previous similar message [ 6385.251056] Lustre: lustre-OST0000: Denying connection for new client 251f9afe-be0a-4522-a288-c3e50cc32311 (at 192.168.204.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 6385.260726] Lustre: Skipped 1 previous similar message [ 6395.490957] Lustre: lustre-OST0000: Denying connection for new client 251f9afe-be0a-4522-a288-c3e50cc32311 (at 192.168.204.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 6395.501355] Lustre: Skipped 1 previous similar message [ 6415.970299] Lustre: lustre-OST0000: Denying connection for new client 251f9afe-be0a-4522-a288-c3e50cc32311 (at 192.168.204.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:31 [ 6415.974978] Lustre: Skipped 3 previous similar messages [ 6447.500216] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 6447.504219] Lustre: 251936:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client c3b5c144-cecf-4f60-89b2-54830e14a9e8@ [ 6447.511679] Lustre: 251936:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 6447.516776] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 6447.529373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12884 to 0x280000401:12905) [ 6453.376178] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 6458.309143] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 6461.164779] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 22:09:16 (1768014556) [ 6462.941320] Lustre: Failing over lustre-OST0000 [ 6462.944405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6462.952126] Lustre: Skipped 56 previous similar messages [ 6462.978972] LustreError: 253172:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6462.982551] LustreError: 253172:0:(obd_class.h:479:obd_check_dev()) Skipped 131 previous similar messages [ 6463.006522] Lustre: server umount lustre-OST0000 complete [ 6464.481740] LustreError: 8375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6464.491901] LustreError: 8375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 239 previous similar messages [ 6477.343910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6477.444302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6477.449186] Lustre: Skipped 18 previous similar messages [ 6477.455296] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6477.459808] Lustre: Skipped 18 previous similar messages [ 6478.762525] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6478.765938] Lustre: Skipped 18 previous similar messages [ 6479.345666] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6479.348063] Lustre: Skipped 18 previous similar messages [ 6479.348183] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6479.351171] Lustre: Skipped 57 previous similar messages [ 6479.569277] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6484.582632] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 22:09:39 (1768014579) [ 6486.416468] Lustre: Failing over lustre-OST0000 [ 6486.483826] Lustre: server umount lustre-OST0000 complete [ 6500.592747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6502.583372] LustreError: 254917:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 6502.587590] LustreError: 254917:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 78 previous similar messages [ 6503.462762] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6509.024206] Lustre: *** cfs_fail_loc=715, val=40*** [ 6518.371084] Lustre: lustre-OST0000: Client 251f9afe-be0a-4522-a288-c3e50cc32311 (at 192.168.204.31@tcp) reconnected, waiting for 3 clients in recovery for 1:25 [ 6518.752159] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768014598/real 1768014598] req@ffff9fc25c9b8000 x1853890989468416/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1768014614 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 6518.768691] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 6524.384218] Lustre: *** cfs_fail_loc=715, val=40*** [ 6524.386953] Lustre: Skipped 2 previous similar messages [ 6534.112739] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:09 [ 6534.118583] Lustre: Skipped 3 previous similar messages [ 6540.256235] Lustre: *** cfs_fail_loc=715, val=40*** [ 6540.258854] Lustre: Skipped 2 previous similar messages [ 6542.632129] LustreError: 254917:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6542.639203] LustreError: 254917:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 78 previous similar messages [ 6544.503775] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6545.301606] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6549.722412] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 22:10:45 (1768014645) [ 6551.641895] Lustre: Failing over lustre-MDT0000 [ 6551.960836] Lustre: server umount lustre-MDT0000 complete [ 6565.651634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6565.709886] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6565.715421] LustreError: Skipped 8 previous similar messages [ 6567.234545] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6570.986064] LustreError: 256361:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6576.608244] Lustre: *** cfs_fail_loc=715, val=80*** [ 6576.610901] Lustre: Skipped 2 previous similar messages [ 6585.973894] Lustre: lustre-MDT0000: Client 251f9afe-be0a-4522-a288-c3e50cc32311 (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 1:25 [ 6585.980897] Lustre: Skipped 1 previous similar message [ 6587.360450] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6592.480296] Lustre: *** cfs_fail_loc=715, val=80*** [ 6592.482965] Lustre: Skipped 1 previous similar message [ 6602.338299] Lustre: lustre-MDT0000: Client 251f9afe-be0a-4522-a288-c3e50cc32311 (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 1:09 [ 6603.744520] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6608.352234] Lustre: *** cfs_fail_loc=715, val=80*** [ 6608.355137] Lustre: Skipped 1 previous similar message [ 6618.722110] Lustre: lustre-MDT0000: Client 251f9afe-be0a-4522-a288-c3e50cc32311 (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 0:52 [ 6619.104825] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6624.736334] Lustre: *** cfs_fail_loc=715, val=80*** [ 6624.739163] Lustre: Skipped 1 previous similar message [ 6634.082113] Lustre: lustre-MDT0000: Client 251f9afe-be0a-4522-a288-c3e50cc32311 (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 0:37 [ 6635.488728] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6650.465834] Lustre: lustre-MDT0000: Client 251f9afe-be0a-4522-a288-c3e50cc32311 (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 0:21 [ 6651.072164] LustreError: 256361:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6651.079877] Lustre: 256361:0:(ldlm_lib.c:2931:target_recovery_thread()) too long recovery - read logs [ 6651.082917] LustreError: dumping log to /tmp/lustre-log.1768014747.256361 [ 6651.150479] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12910 to 0x2c0000401:12929) [ 6651.150652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12919 to 0x280000401:12937) [ 6652.835753] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6653.623450] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6657.969324] Lustre: DEBUG MARKER: == replay-single test complete, duration 6473 sec ======== 22:12:33 (1768014753) [ 6658.638521] Lustre: DEBUG MARKER: === replay-single: start cleanup 22:12:34 (1768014754) === [ 6662.252993] Lustre: DEBUG MARKER: === replay-single: finish cleanup 22:12:37 (1768014757) === [ 6663.099595] Lustre: Failing over lustre-MDT0000 [ 6663.305448] Lustre: server umount lustre-MDT0000 complete [ 6665.186537] LustreError: 144780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768014761 with bad export cookie 9863625530806591765 [ 6665.191938] LustreError: 144780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6679.318158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6680.903411] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6684.675646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12919 to 0x280000401:12969) [ 6684.675669] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12910 to 0x2c0000401:12961) [ 6686.146456] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6686.831893] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6694.881193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6694.883532] LustreError: Skipped 15 previous similar messages [ 6698.730389] Lustre: server umount lustre-MDT0000 complete [ 6701.959679] LustreError: 42018:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768014798 with bad export cookie 9863625530806611869 [ 6701.965852] LustreError: 42018:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6702.066779] Lustre: server umount lustre-MDT0001 complete [ 6715.723044] Lustre: server umount lustre-OST0000 complete [ 6729.448216] Lustre: server umount lustre-OST0001 complete [ 6736.005796] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 6737.157400] Key type lgssc unregistered [ 6737.316291] LNet: 260937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6737.318787] LNetError: 260937:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6737.328331] LNet: Removed LNI 192.168.204.131@tcp [ 6737.757135] Key type .llcrypt unregistered [ 6737.758823] Key type ._llcrypt unregistered